Introduction to Neural Networks - Rene wind Problem Statement¶¶
Problem Statement¶
Build various classification models, tune them and find the best one to identify failures so that the generators can be repaired before failing/breaking to reduce overall maintenance cost.
Nature of predictions:
True Positives (TP) = Failures correctly predicted by model. Hence this will give repair cost.
False Negatives (FN) = Failures not detected by model. Hence the dmaage will be done and it will give replacement cost.
False Positives (FP) = Failures that are wrongly detected by model. Hence resulting in unnecessary inspection cost.
In the end, Inspection cost < Repair cost < Replacement cost.
Data Dictionary¶
Sensor data split into train and test csv files with 40 predictor variables and 1 target variable.
Target variable 1 - 'failure' Target variable 0 - 'no failure'
#Installing libraries
!pip install tensorflow==2.15.0
Collecting tensorflow==2.15.0 Downloading tensorflow-2.15.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.4 kB) Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (1.4.0) Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (1.6.3) Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (25.2.10) Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (0.6.0) Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (0.2.0) Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (3.12.1) Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (18.1.1) Collecting ml-dtypes~=0.2.0 (from tensorflow==2.15.0) Downloading ml_dtypes-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (20 kB) Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (1.26.4) Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (3.4.0) Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (24.2) Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (4.25.6) Requirement already satisfied: setuptools in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (75.1.0) Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (1.17.0) Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (2.5.0) Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (4.12.2) Collecting wrapt<1.15,>=1.11.0 (from tensorflow==2.15.0) Downloading wrapt-1.14.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB) Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (0.37.1) Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.11/dist-packages (from tensorflow==2.15.0) (1.70.0) Collecting tensorboard<2.16,>=2.15 (from tensorflow==2.15.0) Downloading tensorboard-2.15.2-py3-none-any.whl.metadata (1.7 kB) Collecting tensorflow-estimator<2.16,>=2.15.0 (from tensorflow==2.15.0) Downloading tensorflow_estimator-2.15.0-py2.py3-none-any.whl.metadata (1.3 kB) Collecting keras<2.16,>=2.15.0 (from tensorflow==2.15.0) Downloading keras-2.15.0-py3-none-any.whl.metadata (2.4 kB) Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.11/dist-packages (from astunparse>=1.6.0->tensorflow==2.15.0) (0.45.1) Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (2.27.0) Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (1.2.1) Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.7) Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (2.32.3) Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (0.7.2) Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.1.3) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.11/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (5.5.1) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.11/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (0.4.1) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.11/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (4.9) Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.11/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (2.0.0) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.4.1) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (2.3.0) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (2025.1.31) Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.11/dist-packages (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.0.2) Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.11/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (0.6.1) Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.11/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow==2.15.0) (3.2.2) Downloading tensorflow-2.15.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (475.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 475.3/475.3 MB 1.2 MB/s eta 0:00:00 Downloading keras-2.15.0-py3-none-any.whl (1.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 6.1 MB/s eta 0:00:00 Downloading ml_dtypes-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 8.7 MB/s eta 0:00:00 Downloading tensorboard-2.15.2-py3-none-any.whl (5.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 8.5 MB/s eta 0:00:00 Downloading tensorflow_estimator-2.15.0-py2.py3-none-any.whl (441 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 442.0/442.0 kB 12.3 MB/s eta 0:00:00 Downloading wrapt-1.14.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 kB 2.2 MB/s eta 0:00:00 Installing collected packages: wrapt, tensorflow-estimator, ml-dtypes, keras, tensorboard, tensorflow Attempting uninstall: wrapt Found existing installation: wrapt 1.17.2 Uninstalling wrapt-1.17.2: Successfully uninstalled wrapt-1.17.2 Attempting uninstall: ml-dtypes Found existing installation: ml-dtypes 0.4.1 Uninstalling ml-dtypes-0.4.1: Successfully uninstalled ml-dtypes-0.4.1 Attempting uninstall: keras Found existing installation: keras 3.8.0 Uninstalling keras-3.8.0: Successfully uninstalled keras-3.8.0 Attempting uninstall: tensorboard Found existing installation: tensorboard 2.18.0 Uninstalling tensorboard-2.18.0: Successfully uninstalled tensorboard-2.18.0 Attempting uninstall: tensorflow Found existing installation: tensorflow 2.18.0 Uninstalling tensorflow-2.18.0: Successfully uninstalled tensorflow-2.18.0 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. tensorflow-text 2.18.1 requires tensorflow<2.19,>=2.18.0, but you have tensorflow 2.15.0 which is incompatible. tensorstore 0.1.71 requires ml_dtypes>=0.3.1, but you have ml-dtypes 0.2.0 which is incompatible. tf-keras 2.18.0 requires tensorflow<2.19,>=2.18, but you have tensorflow 2.15.0 which is incompatible. Successfully installed keras-2.15.0 ml-dtypes-0.2.0 tensorboard-2.15.2 tensorflow-2.15.0 tensorflow-estimator-2.15.0 wrapt-1.14.1
Importing Libraries¶
#Import necessary libraries for EDA
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
#Sklearn library imports
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, OneHotEncoder,StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.metrics import confusion_matrix, f1_score,accuracy_score, recall_score, precision_score, classification_report, precision_recall_curve
# Time related functions.
import time
#Imports the tensorflow,keras and layers.
import tensorflow
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Dense, Input, Dropout,BatchNormalization
from tensorflow.keras.initializers import HeNormal, GlorotUniform, RandomNormal
from tensorflow.keras.regularizers import l1_l2
from tensorflow.keras import backend
# to suppress unnecessary warnings
import warnings
warnings.filterwarnings("ignore")
# Set the seed using keras.utils.set_random_seed.
tf.keras.utils.set_random_seed(812)
# To reproduce the same output
tf.config.experimental.enable_op_determinism()
Data Overview¶
#Import the data from drive
from google.colab import drive
train_df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/Intro_NN_Renewind/Train.csv')
test_df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/Intro_NN_Renewind/Test.csv')
# Make a copy of the dataset
df_temp = train_df.copy()
#View a glimpse of the Training and Testing dataset
print(train_df.head())
print(test_df.head())
V1 V2 V3 V4 V5 V6 V7 \
0 -4.464606 -4.679129 3.101546 0.506130 -0.221083 -2.032511 -2.910870
1 3.365912 3.653381 0.909671 -1.367528 0.332016 2.358938 0.732600
2 -3.831843 -5.824444 0.634031 -2.418815 -1.773827 1.016824 -2.098941
3 1.618098 1.888342 7.046143 -1.147285 0.083080 -1.529780 0.207309
4 -0.111440 3.872488 -3.758361 -2.982897 3.792714 0.544960 0.205433
V8 V9 V10 ... V32 V33 V34 V35 \
0 0.050714 -1.522351 3.761892 ... 3.059700 -1.690440 2.846296 2.235198
1 -4.332135 0.565695 -0.101080 ... -1.795474 3.032780 -2.467514 1.894599
2 -3.173204 -2.081860 5.392621 ... -0.257101 0.803550 4.086219 2.292138
3 -2.493629 0.344926 2.118578 ... -3.584425 -2.577474 1.363769 0.622714
4 4.848994 -1.854920 -6.220023 ... 8.265896 6.629213 -10.068689 1.222987
V36 V37 V38 V39 V40 Target
0 6.667486 0.443809 -2.369169 2.950578 -3.480324 0
1 -2.297780 -1.731048 5.908837 -0.386345 0.616242 0
2 5.360850 0.351993 2.940021 3.839160 -4.309402 0
3 5.550100 -1.526796 0.138853 3.101430 -1.277378 0
4 -3.229763 1.686909 -2.163896 -3.644622 6.510338 0
[5 rows x 41 columns]
V1 V2 V3 V4 V5 V6 V7 \
0 -0.613489 -3.819640 2.202302 1.300420 -1.184929 -4.495964 -1.835817
1 0.389608 -0.512341 0.527053 -2.576776 -1.016766 2.235112 -0.441301
2 -0.874861 -0.640632 4.084202 -1.590454 0.525855 -1.957592 -0.695367
3 0.238384 1.458607 4.014528 2.534478 1.196987 -3.117330 -0.924035
4 5.828225 2.768260 -1.234530 2.809264 -1.641648 -1.406698 0.568643
V8 V9 V10 ... V32 V33 V34 V35 \
0 4.722989 1.206140 -0.341909 ... 2.291204 -5.411388 0.870073 0.574479
1 -4.405744 -0.332869 1.966794 ... -2.474936 2.493582 0.315165 2.059288
2 1.347309 -1.732348 0.466500 ... -1.318888 -2.997464 0.459664 0.619774
3 0.269493 1.322436 0.702345 ... 3.517918 -3.074085 -0.284220 0.954576
4 0.965043 1.918379 -2.774855 ... 1.773841 -1.501573 -2.226702 4.776830
V36 V37 V38 V39 V40 Target
0 4.157191 1.428093 -10.511342 0.454664 -1.448363 0
1 0.683859 -0.485452 5.128350 1.720744 -1.488235 0
2 5.631504 1.323512 -1.752154 1.808302 1.675748 0
3 3.029331 -1.367198 -3.412140 0.906000 -2.450889 0
4 -6.559698 -0.805551 -0.276007 -3.858207 -0.537694 0
[5 rows x 41 columns]
#View the shape of Training and Testing dataset
print(train_df.shape)
print(test_df.shape)
(20000, 41) (5000, 41)
# View the datatypes of columns in Training and Testing dataset
print(train_df.info())
print(test_df.info())
<class 'pandas.core.frame.DataFrame'> RangeIndex: 20000 entries, 0 to 19999 Data columns (total 41 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 V1 19982 non-null float64 1 V2 19982 non-null float64 2 V3 20000 non-null float64 3 V4 20000 non-null float64 4 V5 20000 non-null float64 5 V6 20000 non-null float64 6 V7 20000 non-null float64 7 V8 20000 non-null float64 8 V9 20000 non-null float64 9 V10 20000 non-null float64 10 V11 20000 non-null float64 11 V12 20000 non-null float64 12 V13 20000 non-null float64 13 V14 20000 non-null float64 14 V15 20000 non-null float64 15 V16 20000 non-null float64 16 V17 20000 non-null float64 17 V18 20000 non-null float64 18 V19 20000 non-null float64 19 V20 20000 non-null float64 20 V21 20000 non-null float64 21 V22 20000 non-null float64 22 V23 20000 non-null float64 23 V24 20000 non-null float64 24 V25 20000 non-null float64 25 V26 20000 non-null float64 26 V27 20000 non-null float64 27 V28 20000 non-null float64 28 V29 20000 non-null float64 29 V30 20000 non-null float64 30 V31 20000 non-null float64 31 V32 20000 non-null float64 32 V33 20000 non-null float64 33 V34 20000 non-null float64 34 V35 20000 non-null float64 35 V36 20000 non-null float64 36 V37 20000 non-null float64 37 V38 20000 non-null float64 38 V39 20000 non-null float64 39 V40 20000 non-null float64 40 Target 20000 non-null int64 dtypes: float64(40), int64(1) memory usage: 6.3 MB None <class 'pandas.core.frame.DataFrame'> RangeIndex: 5000 entries, 0 to 4999 Data columns (total 41 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 V1 4995 non-null float64 1 V2 4994 non-null float64 2 V3 5000 non-null float64 3 V4 5000 non-null float64 4 V5 5000 non-null float64 5 V6 5000 non-null float64 6 V7 5000 non-null float64 7 V8 5000 non-null float64 8 V9 5000 non-null float64 9 V10 5000 non-null float64 10 V11 5000 non-null float64 11 V12 5000 non-null float64 12 V13 5000 non-null float64 13 V14 5000 non-null float64 14 V15 5000 non-null float64 15 V16 5000 non-null float64 16 V17 5000 non-null float64 17 V18 5000 non-null float64 18 V19 5000 non-null float64 19 V20 5000 non-null float64 20 V21 5000 non-null float64 21 V22 5000 non-null float64 22 V23 5000 non-null float64 23 V24 5000 non-null float64 24 V25 5000 non-null float64 25 V26 5000 non-null float64 26 V27 5000 non-null float64 27 V28 5000 non-null float64 28 V29 5000 non-null float64 29 V30 5000 non-null float64 30 V31 5000 non-null float64 31 V32 5000 non-null float64 32 V33 5000 non-null float64 33 V34 5000 non-null float64 34 V35 5000 non-null float64 35 V36 5000 non-null float64 36 V37 5000 non-null float64 37 V38 5000 non-null float64 38 V39 5000 non-null float64 39 V40 5000 non-null float64 40 Target 5000 non-null int64 dtypes: float64(40), int64(1) memory usage: 1.6 MB None
#checking for duplicate values in both Training and Testing dataset
print(train_df.duplicated().sum())
print(test_df.duplicated().sum())
0 0
#Checking for missing values in both Training and Testing dataset
print(train_df.isnull().sum())
print(test_df.isnull().sum())
V1 18 V2 18 V3 0 V4 0 V5 0 V6 0 V7 0 V8 0 V9 0 V10 0 V11 0 V12 0 V13 0 V14 0 V15 0 V16 0 V17 0 V18 0 V19 0 V20 0 V21 0 V22 0 V23 0 V24 0 V25 0 V26 0 V27 0 V28 0 V29 0 V30 0 V31 0 V32 0 V33 0 V34 0 V35 0 V36 0 V37 0 V38 0 V39 0 V40 0 Target 0 dtype: int64 V1 5 V2 6 V3 0 V4 0 V5 0 V6 0 V7 0 V8 0 V9 0 V10 0 V11 0 V12 0 V13 0 V14 0 V15 0 V16 0 V17 0 V18 0 V19 0 V20 0 V21 0 V22 0 V23 0 V24 0 V25 0 V26 0 V27 0 V28 0 V29 0 V30 0 V31 0 V32 0 V33 0 V34 0 V35 0 V36 0 V37 0 V38 0 V39 0 V40 0 Target 0 dtype: int64
#View the rows with missing values in column V1
train_df.loc[train_df['V1'].isnull() == True]
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | Target | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 89 | NaN | -3.961403 | 2.787804 | -4.712526 | -3.007329 | -1.541245 | -0.881148 | 1.476656 | 0.574700 | -1.100884 | ... | -8.326069 | -5.140552 | 1.121314 | -0.305907 | 5.315007 | 3.750044 | -5.631174 | 2.372485 | 2.195956 | 0 |
| 5941 | NaN | 1.008391 | 1.227702 | 5.397082 | 0.064230 | -2.706919 | -2.028368 | 0.534046 | 3.006797 | -2.362238 | ... | 1.869502 | -3.115298 | -0.550197 | 1.713781 | -2.256960 | 0.410992 | -3.434400 | -1.299388 | -1.768734 | 0 |
| 6317 | NaN | -5.205346 | 1.997652 | -3.707913 | -1.042200 | -1.593126 | -2.653309 | 0.852280 | -1.310489 | 2.406924 | ... | 3.074149 | -0.067649 | -0.277521 | 3.196840 | 7.016205 | 1.302334 | -4.580096 | 2.956254 | -2.363150 | 0 |
| 6464 | NaN | 2.146202 | 5.004415 | 4.192063 | 1.427887 | -6.438263 | -0.931339 | 3.794120 | -0.683032 | -0.738941 | ... | 5.231472 | -5.113312 | 1.745687 | 2.587189 | 3.990777 | 0.610716 | -4.273457 | 1.864568 | -3.599079 | 0 |
| 7073 | NaN | 2.534010 | 2.762821 | -1.673718 | -1.942214 | -0.029961 | 0.911323 | -3.199743 | 2.948610 | -0.413229 | ... | -4.887077 | -2.611526 | -1.500807 | 2.036186 | -0.828979 | -1.369591 | 0.572366 | -0.132183 | -0.322007 | 0 |
| 8431 | NaN | -1.398710 | -2.008106 | -1.750341 | 0.932279 | -1.290327 | -0.270476 | 4.458834 | -2.776270 | -1.211766 | ... | 4.560244 | -0.420834 | -2.037313 | 1.109793 | 1.520594 | 2.113872 | -2.252571 | -0.939249 | 2.542411 | 0 |
| 8439 | NaN | -3.840585 | 0.197220 | 4.147789 | 1.151400 | -0.993298 | -4.732363 | 0.558966 | -0.926683 | 0.457914 | ... | 6.818725 | 3.451213 | 0.241818 | 3.215765 | 1.203210 | 1.274857 | -1.921229 | 0.578890 | -2.837521 | 0 |
| 11156 | NaN | -0.666978 | 3.715829 | 4.934000 | 1.667596 | -4.356097 | -2.823137 | 0.373175 | -0.709951 | 2.177428 | ... | 6.663446 | -2.897697 | 3.068461 | 2.486862 | 4.808548 | 0.069305 | -1.215784 | 3.013674 | -5.972586 | 0 |
| 11287 | NaN | -2.561519 | -0.180836 | -7.194814 | -1.043832 | 1.384845 | 1.306093 | 1.559192 | -2.992173 | 1.274543 | ... | -2.531655 | 0.560392 | -1.153884 | -0.019205 | 4.065248 | 0.978880 | -0.571288 | 0.630374 | 3.919467 | 0 |
| 11456 | NaN | 1.299595 | 4.382858 | 1.583219 | -0.076564 | 0.658770 | -1.638530 | -4.814763 | -0.914819 | 2.811808 | ... | 1.772287 | 5.755242 | 1.203739 | 5.663939 | 0.413630 | -2.643934 | 5.529745 | 2.104536 | -4.945350 | 0 |
| 12221 | NaN | -2.326319 | -0.051978 | 0.615063 | -0.895755 | -2.437003 | 0.349826 | 2.092611 | -2.933523 | 2.291272 | ... | 0.134995 | -5.183424 | 5.251667 | 0.716371 | 3.210930 | 1.641985 | 1.543559 | 1.805163 | -2.039510 | 0 |
| 12447 | NaN | 0.752613 | -0.271099 | 1.301204 | 2.038697 | -1.485203 | -0.411939 | 0.980629 | 0.810336 | -0.065120 | ... | 4.410397 | -2.208567 | -1.358706 | -1.725697 | 1.679060 | -0.208564 | -2.335547 | 0.112248 | -0.542931 | 0 |
| 13086 | NaN | 2.056243 | 3.330642 | 2.741497 | 2.783166 | -0.444191 | -2.015376 | -0.887154 | -1.110920 | 0.025289 | ... | 5.112126 | 4.675408 | -1.709632 | 2.429762 | 0.996644 | -1.190509 | 1.207054 | 0.511023 | -0.884200 | 0 |
| 13411 | NaN | 2.704511 | 4.587169 | 1.867930 | 2.050133 | -0.925076 | -1.669496 | -1.653803 | -0.243383 | -0.317316 | ... | 2.527207 | 3.625279 | -1.200200 | 2.328028 | 1.666937 | -0.943228 | 0.946846 | 1.655145 | -1.665439 | 0 |
| 14202 | NaN | 7.038653 | 2.144536 | -3.201788 | 4.112972 | 3.375972 | -1.337179 | -4.546371 | 1.941427 | -5.466593 | ... | 0.157778 | 9.768106 | -10.258190 | 0.513864 | -1.974958 | -0.029436 | 3.127486 | 0.009482 | 4.538125 | 0 |
| 15520 | NaN | 1.382556 | 3.236896 | -3.818363 | -1.917264 | 0.437686 | 1.347540 | -2.036067 | 1.155712 | 0.306502 | ... | -5.414599 | -0.896510 | -1.057864 | 1.417365 | 1.161990 | -1.147123 | -0.048258 | 0.604532 | 0.814557 | 0 |
| 16576 | NaN | 3.933815 | -0.761930 | 2.651889 | 1.753614 | -0.554092 | 1.829107 | -0.105409 | -3.737081 | 1.036776 | ... | 3.486408 | 1.028094 | 2.845747 | 1.744060 | -1.999615 | -0.783041 | 8.698449 | 0.352489 | -2.005397 | 0 |
| 18104 | NaN | 1.492173 | 2.659206 | 0.222784 | -0.303648 | -1.347322 | 0.044309 | -0.159095 | 1.108116 | -0.572670 | ... | -1.007343 | -2.229579 | -0.870845 | 1.299595 | 0.667952 | -0.503349 | -1.485419 | -0.153722 | 0.156501 | 0 |
18 rows × 41 columns
#To view rows with missing values in column V2
train_df.loc[train_df['V2'].isnull() == True]
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | Target | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 613 | -2.048681 | NaN | -1.623885 | -3.324224 | 0.152256 | 0.600157 | -1.812802 | 0.852194 | -1.522600 | 0.211071 | ... | 3.264218 | 2.379064 | -2.457084 | 1.719365 | 2.537010 | 1.701780 | -1.434535 | 0.597365 | 0.739238 | 0 |
| 2236 | -3.760658 | NaN | 0.194954 | -1.637958 | 1.261479 | -1.573947 | -3.685700 | 1.575651 | -0.309823 | -0.137656 | ... | 7.620821 | 1.695061 | -3.956354 | 2.707644 | 4.657387 | 1.619307 | -5.537285 | 1.246650 | -1.162793 | 0 |
| 2508 | -1.430888 | NaN | 0.659576 | -2.876402 | 1.150137 | -0.785760 | -1.560174 | 2.898635 | -2.346989 | -0.217607 | ... | 6.279266 | 3.323914 | -4.047760 | 3.119220 | 3.336260 | 0.603524 | -3.781725 | -0.157478 | 1.503298 | 0 |
| 4653 | 5.465769 | NaN | 4.540947 | -2.916550 | 0.399752 | 2.798925 | 0.029477 | -7.334071 | 1.122874 | 1.695269 | ... | -1.535753 | 4.596212 | -4.103525 | 4.295524 | 0.152672 | -3.726700 | 6.562692 | 0.706452 | -0.461696 | 0 |
| 6810 | -2.631454 | NaN | 2.330188 | 1.090080 | 0.603973 | -1.139383 | -0.690121 | -1.358935 | 0.355568 | -1.189176 | ... | -0.950215 | 0.209717 | 0.448728 | 1.046063 | 0.536937 | 0.763187 | 1.728621 | 1.885821 | -1.701774 | 0 |
| 7788 | -4.203459 | NaN | 2.953868 | 0.584466 | 4.103940 | -0.639211 | -2.810799 | -0.112492 | -1.362768 | -0.800101 | ... | 12.522374 | 9.502488 | -7.152953 | 5.668769 | 1.249833 | -2.158520 | -0.954461 | -0.002385 | -1.546808 | 0 |
| 8483 | -4.484232 | NaN | 1.200644 | -2.042064 | 2.779443 | -0.801748 | -5.403548 | -1.225314 | 1.485831 | -0.974382 | ... | 9.467401 | 4.281421 | -7.588117 | 3.266825 | 5.232311 | 1.278590 | -5.370513 | 1.984130 | -1.643391 | 0 |
| 8894 | 3.263555 | NaN | 8.446574 | -3.253218 | -3.417978 | -2.995838 | -0.669271 | -0.161283 | -0.666870 | 3.133527 | ... | -4.242730 | -3.122680 | 2.522415 | 5.283805 | 7.291310 | -0.867555 | -4.315230 | 3.124488 | -2.393239 | 0 |
| 8947 | -3.793170 | NaN | 0.719610 | 2.306296 | 0.934728 | -0.984321 | 0.504867 | -0.441008 | -2.767177 | 1.734671 | ... | 1.527720 | -0.496910 | 3.789736 | 1.130689 | 0.618278 | -0.111146 | 5.708912 | 1.542366 | -2.481019 | 0 |
| 9362 | 2.662045 | NaN | 2.980068 | 4.430762 | -0.237769 | 0.671919 | 0.380068 | -7.646684 | 4.434754 | -0.746393 | ... | -5.493590 | -1.104656 | 1.224987 | 0.975596 | -4.794411 | -2.269039 | 7.670648 | 0.824983 | -3.929104 | 0 |
| 9425 | -2.354134 | NaN | 2.053893 | 0.811660 | 2.540366 | -0.924875 | -0.208380 | -0.562864 | -0.140210 | -2.146916 | ... | -0.621103 | -0.896509 | -1.181480 | -1.236617 | 1.237120 | 1.228277 | 2.073727 | 1.223874 | 1.472175 | 0 |
| 9848 | -1.763501 | NaN | 2.845012 | -2.753083 | -0.811848 | -0.101166 | -1.382141 | -1.105042 | -0.054339 | 0.159742 | ... | -2.158880 | 1.859682 | -0.337278 | 1.509300 | 3.408411 | 0.922594 | -1.502959 | 2.514666 | -0.793574 | 0 |
| 11637 | -2.270541 | NaN | 1.710061 | 1.157522 | -0.355177 | -5.449480 | -0.786321 | 3.936176 | -1.576138 | 0.800881 | ... | 2.651480 | -8.429332 | 3.511387 | 1.500102 | 5.552380 | 2.588580 | -3.453418 | 2.324339 | -2.760081 | 0 |
| 12339 | -1.663687 | NaN | -0.712286 | -4.346935 | 1.391670 | -0.093951 | -2.163175 | -0.380573 | 0.031191 | -0.658845 | ... | 0.306588 | -2.690990 | -3.111879 | -1.596402 | 5.821108 | 3.462033 | -1.736752 | 2.291092 | 2.240769 | 0 |
| 15913 | 0.768122 | NaN | 5.296110 | 0.043018 | -1.173729 | -2.248575 | 0.956395 | -0.089941 | -0.241678 | -1.061413 | ... | -7.720265 | -4.518617 | 3.182253 | 0.453452 | 2.175494 | 1.261707 | 0.892630 | 2.026732 | 0.632903 | 0 |
| 18342 | -0.928572 | NaN | 2.375506 | -1.236914 | 3.228744 | -2.100088 | -2.189908 | 0.588644 | 1.955973 | -5.008491 | ... | 1.613181 | -1.820569 | -6.664808 | -0.455080 | 3.054891 | 2.935276 | -3.791135 | 0.863011 | 3.335753 | 0 |
| 18343 | -2.377369 | NaN | -0.009173 | -1.471979 | 1.295482 | 0.724894 | -1.122797 | -3.190475 | 3.250575 | -4.861648 | ... | -5.093149 | 0.439355 | -3.167241 | -2.713266 | -0.592845 | 3.229219 | 1.315635 | 2.282838 | 1.151589 | 0 |
| 18907 | -0.119181 | NaN | 3.657612 | -1.231802 | 1.946873 | -0.119089 | 0.652414 | -1.490208 | -0.033631 | -2.556604 | ... | -4.670353 | -0.593916 | -1.650592 | -1.405071 | 1.531267 | 1.079147 | 2.832949 | 1.450781 | 3.232659 | 0 |
18 rows × 41 columns
#since the rows where values are missing are in random order and also the total number of rows where data is missing = 36 which is 0.18 % of total rows (20,000). hence drop the rows that have missing data in column V1 and V2
train_df = train_df.dropna(subset=['V1','V2'])
train_df.isnull().sum()
train_df.shape
(19964, 41)
#since the rows where values are missing are in random order and also the total number of rows where data is missing = 11 which is 0.22 % of total rows (5,000).
#hence drop the rows that have missing data in column V1 and V2 in testing dataset
test_df = test_df.dropna(subset=['V1','V2'])
test_df.isnull().sum()
test_df.shape
(4989, 41)
#To get a statistical summary of the dataset
train_df.describe().T
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| V1 | 19964.0 | -0.271156 | 3.442155 | -11.876451 | -2.737380 | -0.746118 | 1.841050 | 15.493002 |
| V2 | 19964.0 | 0.440507 | 3.150895 | -12.319951 | -1.640580 | 0.470788 | 2.544489 | 13.089269 |
| V3 | 19964.0 | 2.485454 | 3.390736 | -10.708139 | 0.207329 | 2.255786 | 4.569975 | 17.090919 |
| V4 | 19964.0 | -0.082647 | 3.432260 | -15.082052 | -2.345815 | -0.135241 | 2.130757 | 13.236381 |
| V5 | 19964.0 | -0.055028 | 2.105122 | -8.603361 | -1.537210 | -0.103365 | 1.339176 | 8.133797 |
| V6 | 19964.0 | -0.995463 | 2.041211 | -10.227147 | -2.347919 | -1.000852 | 0.380444 | 6.975847 |
| V7 | 19964.0 | -0.879029 | 1.761840 | -7.949681 | -2.031207 | -0.917179 | 0.223695 | 8.006091 |
| V8 | 19964.0 | -0.548384 | 3.296765 | -15.657561 | -2.647433 | -0.390276 | 1.726303 | 11.679495 |
| V9 | 19964.0 | -0.016578 | 2.161033 | -8.596313 | -1.495193 | -0.066895 | 1.409579 | 8.137580 |
| V10 | 19964.0 | -0.012416 | 2.193498 | -9.853957 | -1.412478 | 0.102264 | 1.477754 | 8.108472 |
| V11 | 19964.0 | -1.894280 | 3.124513 | -14.832058 | -3.922404 | -1.920594 | 0.119336 | 11.826433 |
| V12 | 19964.0 | 1.603980 | 2.930702 | -12.948007 | -0.396660 | 1.507160 | 3.571182 | 15.080698 |
| V13 | 19964.0 | 1.580054 | 2.875043 | -13.228247 | -0.224047 | 1.636446 | 3.459854 | 15.419616 |
| V14 | 19964.0 | -0.950604 | 1.789884 | -7.738593 | -2.170457 | -0.957426 | 0.270452 | 5.670664 |
| V15 | 19964.0 | -2.413627 | 3.355168 | -16.416606 | -4.413297 | -2.382285 | -0.358942 | 12.246455 |
| V16 | 19964.0 | -2.925148 | 4.223953 | -20.374158 | -5.638282 | -2.683801 | -0.092976 | 13.583212 |
| V17 | 19964.0 | -0.134968 | 3.345533 | -14.091184 | -2.215393 | -0.015050 | 2.066289 | 16.756432 |
| V18 | 19964.0 | 1.187689 | 2.592117 | -11.643994 | -0.405252 | 0.881894 | 2.569720 | 13.179863 |
| V19 | 19964.0 | 1.183252 | 3.396921 | -13.491784 | -1.047863 | 1.281276 | 3.492482 | 13.237742 |
| V20 | 19964.0 | 0.023871 | 3.669621 | -13.922659 | -2.433116 | 0.033415 | 2.513245 | 16.052339 |
| V21 | 19964.0 | -3.611479 | 3.569582 | -17.956231 | -5.932277 | -3.532459 | -1.265252 | 13.840473 |
| V22 | 19964.0 | 0.951407 | 1.651725 | -10.122095 | -0.118339 | 0.974687 | 2.025396 | 7.409856 |
| V23 | 19964.0 | -0.367829 | 4.032946 | -14.866128 | -3.102770 | -0.263199 | 2.450676 | 14.458734 |
| V24 | 19964.0 | 1.133309 | 3.912658 | -16.387147 | -1.468354 | 0.965578 | 3.544004 | 17.163291 |
| V25 | 19964.0 | -0.001582 | 2.017157 | -8.228266 | -1.364949 | 0.026305 | 1.397890 | 8.223389 |
| V26 | 19964.0 | 1.873105 | 3.434893 | -11.834271 | -0.337863 | 1.950531 | 4.128337 | 16.836410 |
| V27 | 19964.0 | -0.610981 | 4.369482 | -14.904939 | -3.650416 | -0.878398 | 2.189664 | 17.560404 |
| V28 | 19964.0 | -0.882974 | 1.917511 | -9.269489 | -2.170332 | -0.891073 | 0.375977 | 6.527643 |
| V29 | 19964.0 | -0.985055 | 2.684795 | -12.579469 | -2.787329 | -1.177999 | 0.629773 | 10.722055 |
| V30 | 19964.0 | -0.015346 | 3.006321 | -14.796047 | -1.866374 | 0.184650 | 2.037034 | 12.505812 |
| V31 | 19964.0 | 0.486920 | 3.462907 | -13.722760 | -1.818319 | 0.491126 | 2.731102 | 17.255090 |
| V32 | 19964.0 | 0.302523 | 5.501439 | -19.876502 | -3.421328 | 0.049207 | 3.760799 | 23.633187 |
| V33 | 19964.0 | 0.049735 | 3.574591 | -16.898353 | -2.242489 | -0.064830 | 2.254949 | 16.692486 |
| V34 | 19964.0 | -0.461671 | 3.183388 | -17.985094 | -2.136799 | -0.253433 | 1.437620 | 14.358213 |
| V35 | 19964.0 | 2.230852 | 2.938409 | -15.349803 | 0.336096 | 2.101516 | 4.068591 | 15.291065 |
| V36 | 19964.0 | 1.513850 | 3.802518 | -14.833178 | -0.948662 | 1.566526 | 3.983750 | 19.329576 |
| V37 | 19964.0 | 0.010491 | 1.788173 | -5.478350 | -1.257895 | -0.130650 | 1.173907 | 7.467006 |
| V38 | 19964.0 | -0.344184 | 3.948539 | -17.375002 | -2.987116 | -0.315217 | 2.280863 | 15.289923 |
| V39 | 19964.0 | 0.890116 | 1.753959 | -6.438880 | -0.273597 | 0.919204 | 2.057540 | 7.759877 |
| V40 | 19964.0 | -0.876240 | 3.013024 | -11.023935 | -2.943705 | -0.921977 | 1.119780 | 10.654265 |
| Target | 19964.0 | 0.055600 | 0.229154 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.000000 |
# Check for class distribution of the target variable by looking at the count of observations for each class in Training Dataset
class_distribution = train_df['Target'].value_counts(normalize=True) * 100
print(class_distribution)
Target 0 94.439992 1 5.560008 Name: proportion, dtype: float64
# Check for class distribution of the target variable by looking at the count of observations for each class in Testing Dataset
class_distribution = test_df['Target'].value_counts(normalize=True) * 100
print(class_distribution)
Target 0 94.347565 1 5.652435 Name: proportion, dtype: float64
Observations:
Dataset Size:
- The training set contains 20,000 observations, each with 41 columns (40 predictors and 1 target variable).
- The test set includes 5,000 observations, also with 41 columns (40 predictors and 1 target variable).
Data Types:
- Both datasets primarily consist of numerical features (float64), with the target variable being an integer (int64).
Missing Values:
- In the training set, two variables (V1 and V2) have 18 missing values each.
- In the test set, V1 has 5 missing values, and V2 has 6 missing values.
- No missing values are present in the target variable of both datasets.
- Since the rows where values are missing are in random order and also the total number of rows where data is missing = 36 which is 0.18 % of total rows (20,000). Hence dropped the rows that have missing data in column V1 and V2 in Traning dataset.
Data Integrity:
- There are no duplicate rows in either the training or the test dataset,indicating good data integrity.
Class Imbalance:
Both training and testing target variables have imbalanced distribution of the target variables classes where:
- About 94% of the observations belong to the negative class (0), which represents "Non-failure" cases.
- About 5% of the observations belong to the positive class (1), representing "Failure" cases.
EDA - Univariate Analysis¶
#define a function to plot the boxplot and histogram for all numerical columns
def histogram_boxplot(data, feature, figsize=(15,10), kde =True):
fig,(axis_box,axis_hist) = plt.subplots(nrows=2,sharex=True,gridspec_kw={'height_ratios':(0.25,0.75)},figsize=figsize)
#creating 2 subplots
sns.boxplot(data=data,x=feature,ax=axis_box,showmeans=True,color='violet')
sns.histplot(data=data,x=feature,kde=kde,ax=axis_hist,color='red')
axis_hist.axvline(data[feature].mean(),color='green',linestyle='dotted')
axis_hist.axvline(data[feature].median(),color='black',linestyle='dashed')
#Create the plots for all the independant columns
for col in train_df.columns:
if col != 'Target':
histogram_boxplot(train_df,col)
#Code to explain the skewness of the data in detail
from scipy.stats import skew
# Skewness
skewness = train_df.apply(skew)
for column in train_df.columns[:-1]: # Exclude target column
skew_val = skewness[column]
if skew_val < -1:
skew_type = "highly negatively skewed"
elif -1 <= skew_val < -0.5:
skew_type = "moderately negatively skewed"
elif -0.5 <= skew_val < 0.5:
skew_type = "approximately symmetric"
elif 0.5 <= skew_val < 1:
skew_type = "moderately positively skewed"
else:
skew_type = "highly positively skewed"
print(f'The distribution of {column} is {skew_type} with a skewness of {skew_val:.2f}.')
The distribution of V1 is moderately positively skewed with a skewness of 0.54. The distribution of V2 is approximately symmetric with a skewness of -0.04. The distribution of V3 is approximately symmetric with a skewness of 0.32. The distribution of V4 is approximately symmetric with a skewness of 0.10. The distribution of V5 is approximately symmetric with a skewness of 0.13. The distribution of V6 is approximately symmetric with a skewness of -0.03. The distribution of V7 is approximately symmetric with a skewness of 0.18. The distribution of V8 is approximately symmetric with a skewness of -0.26. The distribution of V9 is approximately symmetric with a skewness of 0.10. The distribution of V10 is approximately symmetric with a skewness of -0.25. The distribution of V11 is approximately symmetric with a skewness of 0.04. The distribution of V12 is approximately symmetric with a skewness of 0.09. The distribution of V13 is approximately symmetric with a skewness of -0.18. The distribution of V14 is approximately symmetric with a skewness of 0.00. The distribution of V15 is approximately symmetric with a skewness of -0.06. The distribution of V16 is approximately symmetric with a skewness of -0.21. The distribution of V17 is approximately symmetric with a skewness of -0.14. The distribution of V18 is approximately symmetric with a skewness of 0.30. The distribution of V19 is approximately symmetric with a skewness of -0.14. The distribution of V20 is approximately symmetric with a skewness of -0.03. The distribution of V21 is approximately symmetric with a skewness of -0.01. The distribution of V22 is approximately symmetric with a skewness of -0.17. The distribution of V23 is approximately symmetric with a skewness of -0.11. The distribution of V24 is approximately symmetric with a skewness of 0.22. The distribution of V25 is approximately symmetric with a skewness of -0.09. The distribution of V26 is approximately symmetric with a skewness of -0.09. The distribution of V27 is approximately symmetric with a skewness of 0.31. The distribution of V28 is approximately symmetric with a skewness of 0.04. The distribution of V29 is approximately symmetric with a skewness of 0.34. The distribution of V30 is approximately symmetric with a skewness of -0.36. The distribution of V31 is approximately symmetric with a skewness of 0.01. The distribution of V32 is approximately symmetric with a skewness of 0.24. The distribution of V33 is approximately symmetric with a skewness of 0.14. The distribution of V34 is approximately symmetric with a skewness of -0.36. The distribution of V35 is approximately symmetric with a skewness of 0.08. The distribution of V36 is approximately symmetric with a skewness of -0.03. The distribution of V37 is approximately symmetric with a skewness of 0.32. The distribution of V38 is approximately symmetric with a skewness of -0.02. The distribution of V39 is approximately symmetric with a skewness of -0.05. The distribution of V40 is approximately symmetric with a skewness of 0.10.
Observations:
Independent variables:
The histograms indicate that the data for these variables is approximately normally distributed with mean and median fairly close for most, further suggesting a normal distribution. The boxplots show dots outside of the whiskers, which represent outliers. These are observations that fall significantly higher or lower than the majority of the data.
EDA - Bivariate Analysis¶
# Heatmap to visualise the correlation of numerical variables
plt.figure(figsize=(25,25))
# Generate a mask for the upper triangle
mask = np.triu(np.ones_like(train_df.corr(), dtype=bool))
sns.heatmap(train_df.corr(),mask=mask,annot=True,cmap='coolwarm',vmax=1,vmin=-1,fmt=".2f")
<Axes: >
#Create the scatter plots for all the 40 columns
target_variable = 'Target'
for column in train_df.columns:
if column != target_variable:
plt.figure()
plt.scatter(train_df[column], train_df[target_variable], c=train_df[target_variable], cmap='bwr', marker='o')
plt.xlabel(column)
plt.ylabel(target_variable)
plt.title(f'Scatter Plot of {column} vs. {target_variable}')
plt.show()
#Box plot to see the distribution of columns w.r.t classes in target column
fig, axes = plt.subplots(nrows = 8, ncols = 5, figsize = (14, 24))
axes = axes.flat
for i,col in enumerate(train_df.columns[:-1]):
sns.boxplot(train_df, x = 'Target', y = col, ax = axes[i])
axes[i].set_title(f'{col} by Target', fontsize = 9, fontweight = 'bold', color = 'black')
axes[i].set_xlabel('')
fig.suptitle('Distribution of Numerical Variables by Target', fontsize = 12, fontweight = 'bold', color = 'darkblue', y = 0.93)
fig.tight_layout()
fig.subplots_adjust(top = 0.9)
fig.show()
Observations:
Lots of Highly Correlated Features are there both positive and negative. These may represent redundancy within the dataset which means that one feature can be predicted from the other with a high degree of accuracy. In other words, these features carry similar information or signals about the data.
Some of the strong correlations are seen between - (V11,V6), (V12,V8), (V15,V7), (V16,V8), (V36,V39), (V19,V34), (V24,V32), (V11,V29), (V26,V2)
Columns with strong correlation w.r.t Target variable are - V7, V11, V15, V16, V21, V28, V34.
Data preprocessing¶
#checking the split of target column
train_df.shape
train_df['Target'].value_counts()
#test_df['Target'].value_counts()
| count | |
|---|---|
| Target | |
| 0 | 18854 |
| 1 | 1110 |
#separate the Dependant and Independant variables in Train and Test Dataframe
# Separate features and target for training data
X_train_temp = train_df.drop(columns=['Target'])
y_train_temp = train_df['Target']
# Separate features and target for testing data
X_test = test_df.drop(columns=['Target'])
y_test = test_df['Target']
#Split Training Data into Training and Validation Sets
X_train, X_val, y_train, y_val = train_test_split(X_train_temp, y_train_temp, test_size=0.5, random_state=42)
print("Number of rows in train data =", X_train.shape[0])
print("Number of rows in validation data =", X_val.shape[0])
print("Number of rows in test data =", X_test.shape[0])
Number of rows in train data = 9982 Number of rows in validation data = 9982 Number of rows in test data = 4989
X_train.shape
(9982, 40)
#Normalising the data using standard scalar
#Transform numerical columns in x_train, x_val, and x_test using the fitted scaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_val = scaler.transform(X_val)
X_test = scaler.transform(X_test)
#compute class weights
from sklearn.utils.class_weight import compute_class_weight
class_weights = compute_class_weight(class_weight='balanced',classes=np.unique(y_train),y=y_train)
class_weights = dict(enumerate(class_weights))
Utility Functions¶
#Define a function to compute the different metrics to check the performance of the Neural Network model
def model_performance(model, predictors, target, threshold=0.5):
predictions = model.predict(predictors) > threshold
accuracy = accuracy_score(target, predictions)
recall= recall_score(target, predictions, average = 'weighted')
precision = precision_score(target, predictions, average = 'weighted')
f1 = f1_score(target, predictions, average = 'weighted')
#creating a dataframe of metrics
perf_metrics_df = pd.DataFrame({'Accuracy': accuracy, 'Recall': recall, 'Precision': precision, 'F1 Score': f1}, index=[0])
return perf_metrics_df
columns = ["hidden_layers","neurons_HL","activation_ftn_HL","epochs","batch_size","Batch_Norm","Weight_Ini","dropout","optimizer",
"time(secs)","Train_loss","Validation_loss",
"Train_Precision","Validation_Precision","Train_Recall","Validation_Recall",
"Test_Precision","Test_Recall"]
results = pd.DataFrame(columns=columns)
#Define a fucntion to plot the metric chosen for training and validation data
def plot_training_history(history, metric):
if metric not in history.history or f"val_{metric}" not in history.history:
print(f"The metric {metric} is not available in the history.")
return
plt.figure(figsize=(12, 6))
plt.plot(history.history[metric], label=f"Training {metric}")
plt.plot(history.history[f'val_{metric}'], label=f"Validation {metric}")
plt.title(f"Training and Validation {metric} over Epochs")
plt.xlabel('Epochs')
plt.ylabel(metric.capitalize())
plt.legend(loc='best')
plt.grid(True)
plt.show()
Model Building¶
#Defining the list of metrics to be used for all the models
metrics = [tf.keras.metrics.Precision(name="precision"), tf.keras.metrics.Recall(name="recall")]
Model 0 - Shallow Neural Network to check the linearity of the model¶
- No Hidden layers
- SGD
- Loss function - Binary Cross entropy
- Epochs - 10
- Batch size - Input rows
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model0 = Sequential()
model0.add(Dense(1,input_dim = X_train.shape[1]))
model0.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 1) 41
=================================================================
Total params: 41 (164.00 Byte)
Trainable params: 41 (164.00 Byte)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
#Add the necessary configurations and fit the model
optimizer = keras.optimizers.SGD()
model0.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 10
batch_size = X_train.shape[0]
start = time.time()
history = model0.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val))
end = time.time()
Epoch 1/10 1/1 [==============================] - 1s 724ms/step - loss: 4.6068 - precision: 0.0319 - recall: 0.2228 - val_loss: 4.5683 - val_precision: 0.0339 - val_recall: 0.2170 Epoch 2/10 1/1 [==============================] - 0s 49ms/step - loss: 4.4507 - precision: 0.0298 - recall: 0.2041 - val_loss: 4.3926 - val_precision: 0.0347 - val_recall: 0.2170 Epoch 3/10 1/1 [==============================] - 0s 39ms/step - loss: 4.2698 - precision: 0.0310 - recall: 0.2060 - val_loss: 4.2258 - val_precision: 0.0357 - val_recall: 0.2170 Epoch 4/10 1/1 [==============================] - 0s 34ms/step - loss: 4.1031 - precision: 0.0314 - recall: 0.2041 - val_loss: 4.0429 - val_precision: 0.0367 - val_recall: 0.2170 Epoch 5/10 1/1 [==============================] - 0s 49ms/step - loss: 3.9199 - precision: 0.0320 - recall: 0.2022 - val_loss: 3.7776 - val_precision: 0.0391 - val_recall: 0.2222 Epoch 6/10 1/1 [==============================] - 0s 48ms/step - loss: 3.6104 - precision: 0.0331 - recall: 0.2004 - val_loss: 3.0733 - val_precision: 0.0530 - val_recall: 0.2691 Epoch 7/10 1/1 [==============================] - 0s 45ms/step - loss: 2.9496 - precision: 0.0482 - recall: 0.2584 - val_loss: 2.7615 - val_precision: 0.0550 - val_recall: 0.2587 Epoch 8/10 1/1 [==============================] - 0s 32ms/step - loss: 2.6542 - precision: 0.0517 - recall: 0.2566 - val_loss: 2.3993 - val_precision: 0.0566 - val_recall: 0.2448 Epoch 9/10 1/1 [==============================] - 0s 33ms/step - loss: 2.3250 - precision: 0.0517 - recall: 0.2303 - val_loss: 2.0632 - val_precision: 0.0578 - val_recall: 0.2292 Epoch 10/10 1/1 [==============================] - 0s 47ms/step - loss: 2.0315 - precision: 0.0543 - recall: 0.2191 - val_loss: 1.6960 - val_precision: 0.0569 - val_recall: 0.1892
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model0, X_test, y_test)
output
156/156 [==============================] - 0s 1ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.764682 | 0.764682 | 0.890782 | 0.820497 |
results.loc[0] = ['-','-','-',epochs,batch_size,'-','-','-',"SGD",end-start,history.history['loss'][-1],history.history['val_loss'][-1],history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.05686 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
Model 1 - Shallow Network¶
- No Hidden layers
- SGD with learning rate - 0.01
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - Input rows
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model1 = Sequential()
model1.add(Dense(1,input_dim = X_train.shape[1]))
model1.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 1) 41
=================================================================
Total params: 41 (164.00 Byte)
Trainable params: 41 (164.00 Byte)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
#Add the necessary configurations and fit the model
optimizer = keras.optimizers.SGD(learning_rate=0.01)
model1.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = X_train.shape[0]
start = time.time()
history = model1.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val))
end = time.time()
Epoch 1/25 1/1 [==============================] - 1s 651ms/step - loss: 4.3678 - precision: 0.0436 - recall: 0.2054 - val_loss: 4.1789 - val_precision: 0.0357 - val_recall: 0.1979 Epoch 2/25 1/1 [==============================] - 0s 45ms/step - loss: 4.0851 - precision: 0.0299 - recall: 0.1760 - val_loss: 3.9125 - val_precision: 0.0370 - val_recall: 0.1962 Epoch 3/25 1/1 [==============================] - 0s 32ms/step - loss: 3.7993 - precision: 0.0306 - recall: 0.1723 - val_loss: 3.6912 - val_precision: 0.0390 - val_recall: 0.1979 Epoch 4/25 1/1 [==============================] - 0s 48ms/step - loss: 3.6145 - precision: 0.0339 - recall: 0.1835 - val_loss: 3.4839 - val_precision: 0.0387 - val_recall: 0.1892 Epoch 5/25 1/1 [==============================] - 0s 48ms/step - loss: 3.4107 - precision: 0.0352 - recall: 0.1835 - val_loss: 3.2554 - val_precision: 0.0397 - val_recall: 0.1858 Epoch 6/25 1/1 [==============================] - 0s 48ms/step - loss: 3.1711 - precision: 0.0357 - recall: 0.1760 - val_loss: 3.0874 - val_precision: 0.0391 - val_recall: 0.1753 Epoch 7/25 1/1 [==============================] - 0s 54ms/step - loss: 3.0135 - precision: 0.0370 - recall: 0.1742 - val_loss: 2.9178 - val_precision: 0.0387 - val_recall: 0.1632 Epoch 8/25 1/1 [==============================] - 0s 48ms/step - loss: 2.8593 - precision: 0.0364 - recall: 0.1629 - val_loss: 2.4811 - val_precision: 0.0409 - val_recall: 0.1545 Epoch 9/25 1/1 [==============================] - 0s 49ms/step - loss: 2.4252 - precision: 0.0403 - recall: 0.1610 - val_loss: 2.0866 - val_precision: 0.0400 - val_recall: 0.1337 Epoch 10/25 1/1 [==============================] - 0s 50ms/step - loss: 2.0360 - precision: 0.0418 - recall: 0.1479 - val_loss: 1.8209 - val_precision: 0.0668 - val_recall: 0.2274 Epoch 11/25 1/1 [==============================] - 0s 51ms/step - loss: 1.7439 - precision: 0.0670 - recall: 0.2472 - val_loss: 1.2730 - val_precision: 0.0780 - val_recall: 0.2014 Epoch 12/25 1/1 [==============================] - 0s 36ms/step - loss: 1.2054 - precision: 0.0833 - recall: 0.2285 - val_loss: 0.9071 - val_precision: 0.0980 - val_recall: 0.1736 Epoch 13/25 1/1 [==============================] - 0s 49ms/step - loss: 0.8691 - precision: 0.1071 - recall: 0.2116 - val_loss: 0.7955 - val_precision: 0.1108 - val_recall: 0.1597 Epoch 14/25 1/1 [==============================] - 0s 48ms/step - loss: 0.7535 - precision: 0.1203 - recall: 0.1873 - val_loss: 0.7109 - val_precision: 0.1326 - val_recall: 0.1250 Epoch 15/25 1/1 [==============================] - 0s 47ms/step - loss: 0.6392 - precision: 0.1635 - recall: 0.1592 - val_loss: 0.6625 - val_precision: 0.1791 - val_recall: 0.1458 Epoch 16/25 1/1 [==============================] - 0s 48ms/step - loss: 0.5993 - precision: 0.1892 - recall: 0.1648 - val_loss: 0.6124 - val_precision: 0.2131 - val_recall: 0.1302 Epoch 17/25 1/1 [==============================] - 0s 51ms/step - loss: 0.5657 - precision: 0.2622 - recall: 0.1610 - val_loss: 0.5874 - val_precision: 0.2618 - val_recall: 0.1250 Epoch 18/25 1/1 [==============================] - 0s 56ms/step - loss: 0.5528 - precision: 0.3142 - recall: 0.1536 - val_loss: 0.5705 - val_precision: 0.2992 - val_recall: 0.1267 Epoch 19/25 1/1 [==============================] - 0s 65ms/step - loss: 0.5382 - precision: 0.3591 - recall: 0.1479 - val_loss: 0.5544 - val_precision: 0.3217 - val_recall: 0.1285 Epoch 20/25 1/1 [==============================] - 0s 62ms/step - loss: 0.5232 - precision: 0.3738 - recall: 0.1498 - val_loss: 0.5300 - val_precision: 0.3410 - val_recall: 0.1545 Epoch 21/25 1/1 [==============================] - 0s 70ms/step - loss: 0.5007 - precision: 0.3702 - recall: 0.1816 - val_loss: 0.5180 - val_precision: 0.3771 - val_recall: 0.1545 Epoch 22/25 1/1 [==============================] - 0s 62ms/step - loss: 0.4903 - precision: 0.3983 - recall: 0.1760 - val_loss: 0.5087 - val_precision: 0.4055 - val_recall: 0.1528 Epoch 23/25 1/1 [==============================] - 0s 44ms/step - loss: 0.4820 - precision: 0.4234 - recall: 0.1760 - val_loss: 0.4829 - val_precision: 0.4123 - val_recall: 0.1632 Epoch 24/25 1/1 [==============================] - 0s 44ms/step - loss: 0.4549 - precision: 0.4534 - recall: 0.2097 - val_loss: 0.4479 - val_precision: 0.3839 - val_recall: 0.2726 Epoch 25/25 1/1 [==============================] - 0s 43ms/step - loss: 0.4251 - precision: 0.4028 - recall: 0.3184 - val_loss: 0.4306 - val_precision: 0.4455 - val_recall: 0.2483
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model1, X_test, y_test)
output
156/156 [==============================] - 0s 2ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.940469 | 0.940469 | 0.928051 | 0.932551 |
results.loc[1] = ['-','-','-',epochs,batch_size,'-','-','-',"SGD_LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
Model 2 - Shallow Network¶
- No Hidden layers
- SGD with learning rate - 0.01
- Loss function - Binary Cross entropy
- Epochs - 50
- Batch size - Input rows
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model2 = Sequential()
model2.add(Dense(1,input_dim = X_train.shape[1]))
model2.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 1) 41
=================================================================
Total params: 41 (164.00 Byte)
Trainable params: 41 (164.00 Byte)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
#Add the necessary configurations and fit the model
optimizer = keras.optimizers.SGD(learning_rate=0.01)
model2.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 50
batch_size = X_train.shape[0]
start = time.time()
history = model2.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val))
end = time.time()
Epoch 1/50 1/1 [==============================] - 1s 662ms/step - loss: 3.3087 - precision: 0.1083 - recall: 0.3550 - val_loss: 3.0946 - val_precision: 0.0786 - val_recall: 0.4306 Epoch 2/50 1/1 [==============================] - 0s 34ms/step - loss: 3.0992 - precision: 0.0772 - recall: 0.4588 - val_loss: 2.8564 - val_precision: 0.0818 - val_recall: 0.4253 Epoch 3/50 1/1 [==============================] - 0s 34ms/step - loss: 2.8604 - precision: 0.0783 - recall: 0.4457 - val_loss: 2.5182 - val_precision: 0.0859 - val_recall: 0.4045 Epoch 4/50 1/1 [==============================] - 0s 48ms/step - loss: 2.4603 - precision: 0.0867 - recall: 0.4513 - val_loss: 2.0985 - val_precision: 0.0795 - val_recall: 0.3281 Epoch 5/50 1/1 [==============================] - 0s 51ms/step - loss: 2.0346 - precision: 0.0785 - recall: 0.3577 - val_loss: 1.7005 - val_precision: 0.0802 - val_recall: 0.2847 Epoch 6/50 1/1 [==============================] - 0s 40ms/step - loss: 1.6505 - precision: 0.0802 - recall: 0.3165 - val_loss: 1.3120 - val_precision: 0.0825 - val_recall: 0.2413 Epoch 7/50 1/1 [==============================] - 0s 32ms/step - loss: 1.2481 - precision: 0.0895 - recall: 0.2809 - val_loss: 1.0091 - val_precision: 0.0954 - val_recall: 0.2083 Epoch 8/50 1/1 [==============================] - 0s 48ms/step - loss: 0.9295 - precision: 0.1018 - recall: 0.2378 - val_loss: 0.8807 - val_precision: 0.1058 - val_recall: 0.1840 Epoch 9/50 1/1 [==============================] - 0s 48ms/step - loss: 0.8038 - precision: 0.1121 - recall: 0.2116 - val_loss: 0.7803 - val_precision: 0.1277 - val_recall: 0.1545 Epoch 10/50 1/1 [==============================] - 0s 47ms/step - loss: 0.6883 - precision: 0.1336 - recall: 0.1667 - val_loss: 0.7438 - val_precision: 0.1926 - val_recall: 0.1181 Epoch 11/50 1/1 [==============================] - 0s 31ms/step - loss: 0.6692 - precision: 0.2087 - recall: 0.1348 - val_loss: 0.7204 - val_precision: 0.2152 - val_recall: 0.1233 Epoch 12/50 1/1 [==============================] - 0s 47ms/step - loss: 0.6534 - precision: 0.2355 - recall: 0.1442 - val_loss: 0.7030 - val_precision: 0.2403 - val_recall: 0.1285 Epoch 13/50 1/1 [==============================] - 0s 33ms/step - loss: 0.6393 - precision: 0.2576 - recall: 0.1423 - val_loss: 0.6856 - val_precision: 0.2446 - val_recall: 0.1389 Epoch 14/50 1/1 [==============================] - 0s 53ms/step - loss: 0.6242 - precision: 0.2603 - recall: 0.1536 - val_loss: 0.6807 - val_precision: 0.2572 - val_recall: 0.1389 Epoch 15/50 1/1 [==============================] - 0s 49ms/step - loss: 0.6168 - precision: 0.2785 - recall: 0.1554 - val_loss: 0.6744 - val_precision: 0.2746 - val_recall: 0.1406 Epoch 16/50 1/1 [==============================] - 0s 35ms/step - loss: 0.6126 - precision: 0.2927 - recall: 0.1573 - val_loss: 0.6666 - val_precision: 0.2800 - val_recall: 0.1458 Epoch 17/50 1/1 [==============================] - 0s 49ms/step - loss: 0.6061 - precision: 0.2969 - recall: 0.1629 - val_loss: 0.6620 - val_precision: 0.2911 - val_recall: 0.1476 Epoch 18/50 1/1 [==============================] - 0s 47ms/step - loss: 0.6017 - precision: 0.3063 - recall: 0.1629 - val_loss: 0.6626 - val_precision: 0.3000 - val_recall: 0.1458 Epoch 19/50 1/1 [==============================] - 0s 33ms/step - loss: 0.5971 - precision: 0.3223 - recall: 0.1648 - val_loss: 0.6568 - val_precision: 0.3060 - val_recall: 0.1493 Epoch 20/50 1/1 [==============================] - 0s 46ms/step - loss: 0.5931 - precision: 0.3248 - recall: 0.1667 - val_loss: 0.6522 - val_precision: 0.3246 - val_recall: 0.1510 Epoch 21/50 1/1 [==============================] - 0s 34ms/step - loss: 0.5948 - precision: 0.3448 - recall: 0.1685 - val_loss: 0.6510 - val_precision: 0.3400 - val_recall: 0.1476 Epoch 22/50 1/1 [==============================] - 0s 34ms/step - loss: 0.5919 - precision: 0.3644 - recall: 0.1685 - val_loss: 0.6479 - val_precision: 0.3517 - val_recall: 0.1441 Epoch 23/50 1/1 [==============================] - 0s 32ms/step - loss: 0.5896 - precision: 0.3836 - recall: 0.1667 - val_loss: 0.6463 - val_precision: 0.3689 - val_recall: 0.1441 Epoch 24/50 1/1 [==============================] - 0s 34ms/step - loss: 0.5867 - precision: 0.4018 - recall: 0.1648 - val_loss: 0.6370 - val_precision: 0.3636 - val_recall: 0.1528 Epoch 25/50 1/1 [==============================] - 0s 31ms/step - loss: 0.5816 - precision: 0.4064 - recall: 0.1667 - val_loss: 0.5825 - val_precision: 0.3199 - val_recall: 0.2622 Epoch 26/50 1/1 [==============================] - 0s 31ms/step - loss: 0.5355 - precision: 0.3150 - recall: 0.2678 - val_loss: 0.5710 - val_precision: 0.3364 - val_recall: 0.2535 Epoch 27/50 1/1 [==============================] - 0s 47ms/step - loss: 0.5302 - precision: 0.3293 - recall: 0.2547 - val_loss: 0.5703 - val_precision: 0.3681 - val_recall: 0.2448 Epoch 28/50 1/1 [==============================] - 0s 39ms/step - loss: 0.5238 - precision: 0.3681 - recall: 0.2378 - val_loss: 0.5832 - val_precision: 0.4569 - val_recall: 0.2118 Epoch 29/50 1/1 [==============================] - 0s 56ms/step - loss: 0.5289 - precision: 0.4496 - recall: 0.2004 - val_loss: 0.5735 - val_precision: 0.4561 - val_recall: 0.2257 Epoch 30/50 1/1 [==============================] - 0s 44ms/step - loss: 0.5229 - precision: 0.4618 - recall: 0.2154 - val_loss: 0.5677 - val_precision: 0.4760 - val_recall: 0.2240 Epoch 31/50 1/1 [==============================] - 0s 42ms/step - loss: 0.5200 - precision: 0.4733 - recall: 0.2154 - val_loss: 0.5676 - val_precision: 0.4727 - val_recall: 0.2257 Epoch 32/50 1/1 [==============================] - 0s 32ms/step - loss: 0.5215 - precision: 0.4528 - recall: 0.2154 - val_loss: 0.5639 - val_precision: 0.4851 - val_recall: 0.2257 Epoch 33/50 1/1 [==============================] - 0s 35ms/step - loss: 0.5174 - precision: 0.4634 - recall: 0.2135 - val_loss: 0.5609 - val_precision: 0.4925 - val_recall: 0.2274 Epoch 34/50 1/1 [==============================] - 0s 38ms/step - loss: 0.5133 - precision: 0.4835 - recall: 0.2191 - val_loss: 0.5589 - val_precision: 0.5058 - val_recall: 0.2274 Epoch 35/50 1/1 [==============================] - 0s 34ms/step - loss: 0.5102 - precision: 0.4874 - recall: 0.2172 - val_loss: 0.5558 - val_precision: 0.5219 - val_recall: 0.2274 Epoch 36/50 1/1 [==============================] - 0s 50ms/step - loss: 0.5056 - precision: 0.5043 - recall: 0.2172 - val_loss: 0.5285 - val_precision: 0.4379 - val_recall: 0.2691 Epoch 37/50 1/1 [==============================] - 0s 55ms/step - loss: 0.4790 - precision: 0.4315 - recall: 0.2715 - val_loss: 0.5305 - val_precision: 0.4840 - val_recall: 0.2622 Epoch 38/50 1/1 [==============================] - 0s 52ms/step - loss: 0.4782 - precision: 0.4628 - recall: 0.2566 - val_loss: 0.5272 - val_precision: 0.5119 - val_recall: 0.2622 Epoch 39/50 1/1 [==============================] - 0s 51ms/step - loss: 0.4726 - precision: 0.4759 - recall: 0.2584 - val_loss: 0.5227 - val_precision: 0.4935 - val_recall: 0.2622 Epoch 40/50 1/1 [==============================] - 0s 50ms/step - loss: 0.4676 - precision: 0.4667 - recall: 0.2622 - val_loss: 0.5207 - val_precision: 0.5154 - val_recall: 0.2622 Epoch 41/50 1/1 [==============================] - 0s 48ms/step - loss: 0.4629 - precision: 0.4879 - recall: 0.2640 - val_loss: 0.5075 - val_precision: 0.4728 - val_recall: 0.2865 Epoch 42/50 1/1 [==============================] - 0s 48ms/step - loss: 0.4481 - precision: 0.4890 - recall: 0.2921 - val_loss: 0.5074 - val_precision: 0.5192 - val_recall: 0.2812 Epoch 43/50 1/1 [==============================] - 0s 50ms/step - loss: 0.4473 - precision: 0.5033 - recall: 0.2828 - val_loss: 0.5032 - val_precision: 0.5281 - val_recall: 0.2778 Epoch 44/50 1/1 [==============================] - 0s 48ms/step - loss: 0.4429 - precision: 0.5190 - recall: 0.2809 - val_loss: 0.5005 - val_precision: 0.5408 - val_recall: 0.2760 Epoch 45/50 1/1 [==============================] - 0s 31ms/step - loss: 0.4424 - precision: 0.5267 - recall: 0.2772 - val_loss: 0.4973 - val_precision: 0.5077 - val_recall: 0.2847 Epoch 46/50 1/1 [==============================] - 0s 33ms/step - loss: 0.4381 - precision: 0.5201 - recall: 0.2903 - val_loss: 0.4944 - val_precision: 0.5482 - val_recall: 0.2865 Epoch 47/50 1/1 [==============================] - 0s 34ms/step - loss: 0.4339 - precision: 0.5401 - recall: 0.2903 - val_loss: 0.4948 - val_precision: 0.5186 - val_recall: 0.2899 Epoch 48/50 1/1 [==============================] - 0s 50ms/step - loss: 0.4403 - precision: 0.5356 - recall: 0.2959 - val_loss: 0.4946 - val_precision: 0.5544 - val_recall: 0.2743 Epoch 49/50 1/1 [==============================] - 0s 42ms/step - loss: 0.4365 - precision: 0.5568 - recall: 0.2846 - val_loss: 0.4945 - val_precision: 0.5843 - val_recall: 0.2708 Epoch 50/50 1/1 [==============================] - 0s 40ms/step - loss: 0.4345 - precision: 0.5660 - recall: 0.2809 - val_loss: 0.4932 - val_precision: 0.5985 - val_recall: 0.2691
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model2, X_test, y_test)
output
156/156 [==============================] - 0s 1ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.951694 | 0.951694 | 0.942613 | 0.943551 |
results.loc[2] = ['-','-','-',epochs,batch_size,'-','-','-',"SGD-LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
Model 3 - Shallow Network¶
- No Hidden layers
- SGD with learning rate - 0.01
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - 32
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model3 = Sequential()
model3.add(Dense(1,input_dim = X_train.shape[1]))
model3.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 1) 41
=================================================================
Total params: 41 (164.00 Byte)
Trainable params: 41 (164.00 Byte)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
#Add the necessary configurations and fit the model
optimizer = keras.optimizers.SGD(learning_rate=0.01)
model3.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model3.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val))
end = time.time()
Epoch 1/25 312/312 [==============================] - 2s 4ms/step - loss: 3.5195 - precision: 0.1445 - recall: 0.3946 - val_loss: 1.1456 - val_precision: 0.3056 - val_recall: 0.5747 Epoch 2/25 312/312 [==============================] - 1s 4ms/step - loss: 0.7386 - precision: 0.3907 - recall: 0.3446 - val_loss: 0.5384 - val_precision: 0.7912 - val_recall: 0.3420 Epoch 3/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4587 - precision: 0.6930 - recall: 0.4438 - val_loss: 0.6193 - val_precision: 0.5840 - val_recall: 0.2413 Epoch 4/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4201 - precision: 0.7864 - recall: 0.4345 - val_loss: 0.5043 - val_precision: 0.8421 - val_recall: 0.3611 Epoch 5/25 312/312 [==============================] - 1s 4ms/step - loss: 3.8959 - precision: 0.0889 - recall: 0.4213 - val_loss: 5.9308 - val_precision: 0.0569 - val_recall: 0.3767 Epoch 6/25 312/312 [==============================] - 1s 4ms/step - loss: 5.6630 - precision: 0.0700 - recall: 0.4925 - val_loss: 5.5041 - val_precision: 0.0736 - val_recall: 0.4635 Epoch 7/25 312/312 [==============================] - 1s 4ms/step - loss: 5.4921 - precision: 0.0717 - recall: 0.4906 - val_loss: 5.4136 - val_precision: 0.0697 - val_recall: 0.4288 Epoch 8/25 312/312 [==============================] - 1s 3ms/step - loss: 5.4270 - precision: 0.0728 - recall: 0.4906 - val_loss: 5.2505 - val_precision: 0.0759 - val_recall: 0.4549 Epoch 9/25 312/312 [==============================] - 1s 3ms/step - loss: 5.2429 - precision: 0.0768 - recall: 0.5056 - val_loss: 5.1243 - val_precision: 0.0753 - val_recall: 0.4392 Epoch 10/25 312/312 [==============================] - 1s 4ms/step - loss: 5.0941 - precision: 0.0763 - recall: 0.4831 - val_loss: 5.0282 - val_precision: 0.0727 - val_recall: 0.4097 Epoch 11/25 312/312 [==============================] - 1s 4ms/step - loss: 4.9839 - precision: 0.0755 - recall: 0.4663 - val_loss: 4.9481 - val_precision: 0.0724 - val_recall: 0.4010 Epoch 12/25 312/312 [==============================] - 1s 3ms/step - loss: 4.8353 - precision: 0.0772 - recall: 0.4625 - val_loss: 4.7934 - val_precision: 0.0755 - val_recall: 0.4062 Epoch 13/25 312/312 [==============================] - 1s 4ms/step - loss: 4.7986 - precision: 0.0634 - recall: 0.3633 - val_loss: 4.4633 - val_precision: 0.0561 - val_recall: 0.2656 Epoch 14/25 312/312 [==============================] - 1s 4ms/step - loss: 4.4105 - precision: 0.0553 - recall: 0.2828 - val_loss: 4.1134 - val_precision: 0.0560 - val_recall: 0.2413 Epoch 15/25 312/312 [==============================] - 1s 4ms/step - loss: 4.6336 - precision: 0.0454 - recall: 0.2397 - val_loss: 4.5869 - val_precision: 0.0467 - val_recall: 0.2240 Epoch 16/25 312/312 [==============================] - 1s 3ms/step - loss: 4.4598 - precision: 0.0492 - recall: 0.2509 - val_loss: 4.2531 - val_precision: 0.0524 - val_recall: 0.2326 Epoch 17/25 312/312 [==============================] - 1s 4ms/step - loss: 4.1858 - precision: 0.0506 - recall: 0.2397 - val_loss: 3.8781 - val_precision: 0.0465 - val_recall: 0.1823 Epoch 18/25 312/312 [==============================] - 1s 4ms/step - loss: 3.8732 - precision: 0.0405 - recall: 0.1704 - val_loss: 3.6943 - val_precision: 0.0454 - val_recall: 0.1667 Epoch 19/25 312/312 [==============================] - 1s 4ms/step - loss: 3.4705 - precision: 0.0396 - recall: 0.1461 - val_loss: 3.2010 - val_precision: 0.0483 - val_recall: 0.1476 Epoch 20/25 312/312 [==============================] - 1s 4ms/step - loss: 3.0757 - precision: 0.0333 - recall: 0.1030 - val_loss: 2.7556 - val_precision: 0.0308 - val_recall: 0.0747 Epoch 21/25 312/312 [==============================] - 1s 4ms/step - loss: 2.6314 - precision: 0.0264 - recall: 0.0655 - val_loss: 2.3652 - val_precision: 0.0317 - val_recall: 0.0608 Epoch 22/25 312/312 [==============================] - 1s 3ms/step - loss: 2.0667 - precision: 0.0273 - recall: 0.0487 - val_loss: 3.3379 - val_precision: 0.0246 - val_recall: 0.0747 Epoch 23/25 312/312 [==============================] - 1s 4ms/step - loss: 3.1269 - precision: 0.0190 - recall: 0.0581 - val_loss: 3.1142 - val_precision: 0.0233 - val_recall: 0.0642 Epoch 24/25 312/312 [==============================] - 1s 3ms/step - loss: 2.7097 - precision: 0.0193 - recall: 0.0487 - val_loss: 2.5656 - val_precision: 0.0221 - val_recall: 0.0469 Epoch 25/25 312/312 [==============================] - 1s 3ms/step - loss: 2.3790 - precision: 0.0227 - recall: 0.0468 - val_loss: 2.3607 - val_precision: 0.0235 - val_recall: 0.0434
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model3, X_test, y_test)
output
156/156 [==============================] - 0s 1ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.844257 | 0.844257 | 0.888372 | 0.865524 |
results.loc[3] = ['-','-','-',epochs,batch_size,'-','-','-',"SGD-LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
Model 4¶
- one Hidden layer - ReLu activation function - 80 nodes
- SGD with learning rate - 0.01
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - 32
- Class weights taken into account
- Output layer - Sigmoid Activaion Function
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model4 = Sequential()
model4.add(Dense(80,activation="relu",input_dim=X_train.shape[1]))
model4.add(Dense(1,activation="sigmoid"))
model4.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 80) 3280
dense_1 (Dense) (None, 1) 81
=================================================================
Total params: 3361 (13.13 KB)
Trainable params: 3361 (13.13 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
#Add the necessary configurations and fit the model
optimizer = keras.optimizers.SGD(learning_rate=0.01)
model4.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model4.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()
Epoch 1/25 312/312 [==============================] - 2s 4ms/step - loss: 0.4483 - precision: 0.1280 - recall: 0.4261 - val_loss: 0.3979 - val_precision: 0.2378 - val_recall: 0.8889 Epoch 2/25 312/312 [==============================] - 1s 3ms/step - loss: 0.3613 - precision: 0.2605 - recall: 0.8727 - val_loss: 0.3351 - val_precision: 0.3088 - val_recall: 0.9028 Epoch 3/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3284 - precision: 0.3142 - recall: 0.8839 - val_loss: 0.2721 - val_precision: 0.4046 - val_recall: 0.9097 Epoch 4/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3068 - precision: 0.3636 - recall: 0.8858 - val_loss: 0.2523 - val_precision: 0.4454 - val_recall: 0.9062 Epoch 5/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2905 - precision: 0.4041 - recall: 0.8764 - val_loss: 0.2433 - val_precision: 0.4815 - val_recall: 0.9062 Epoch 6/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2775 - precision: 0.4225 - recall: 0.8876 - val_loss: 0.2304 - val_precision: 0.5188 - val_recall: 0.9080 Epoch 7/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2670 - precision: 0.4606 - recall: 0.8876 - val_loss: 0.2185 - val_precision: 0.5365 - val_recall: 0.9062 Epoch 8/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2576 - precision: 0.4887 - recall: 0.8876 - val_loss: 0.2414 - val_precision: 0.4726 - val_recall: 0.9132 Epoch 9/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2505 - precision: 0.4823 - recall: 0.8914 - val_loss: 0.1990 - val_precision: 0.6118 - val_recall: 0.9028 Epoch 10/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2448 - precision: 0.5197 - recall: 0.8895 - val_loss: 0.2009 - val_precision: 0.5794 - val_recall: 0.9062 Epoch 11/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2394 - precision: 0.5231 - recall: 0.8895 - val_loss: 0.1893 - val_precision: 0.6159 - val_recall: 0.8993 Epoch 12/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2345 - precision: 0.5326 - recall: 0.8876 - val_loss: 0.1843 - val_precision: 0.6330 - val_recall: 0.8924 Epoch 13/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2302 - precision: 0.5570 - recall: 0.8876 - val_loss: 0.1935 - val_precision: 0.6103 - val_recall: 0.9028 Epoch 14/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2258 - precision: 0.5548 - recall: 0.8914 - val_loss: 0.1991 - val_precision: 0.5763 - val_recall: 0.9045 Epoch 15/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2212 - precision: 0.5504 - recall: 0.8895 - val_loss: 0.1776 - val_precision: 0.6318 - val_recall: 0.9028 Epoch 16/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2171 - precision: 0.5633 - recall: 0.8914 - val_loss: 0.1631 - val_precision: 0.6759 - val_recall: 0.8906 Epoch 17/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2161 - precision: 0.5812 - recall: 0.8914 - val_loss: 0.1894 - val_precision: 0.5957 - val_recall: 0.9080 Epoch 18/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2110 - precision: 0.5962 - recall: 0.8933 - val_loss: 0.2191 - val_precision: 0.4916 - val_recall: 0.9097 Epoch 19/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2080 - precision: 0.5803 - recall: 0.8933 - val_loss: 0.2003 - val_precision: 0.5349 - val_recall: 0.9045 Epoch 20/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2064 - precision: 0.5770 - recall: 0.8914 - val_loss: 0.1758 - val_precision: 0.6225 - val_recall: 0.9045 Epoch 21/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2035 - precision: 0.6028 - recall: 0.8951 - val_loss: 0.1727 - val_precision: 0.6283 - val_recall: 0.9010 Epoch 22/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2004 - precision: 0.6003 - recall: 0.8970 - val_loss: 0.1820 - val_precision: 0.6002 - val_recall: 0.9045 Epoch 23/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1990 - precision: 0.6020 - recall: 0.9007 - val_loss: 0.1734 - val_precision: 0.6311 - val_recall: 0.9028 Epoch 24/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1951 - precision: 0.5877 - recall: 0.8970 - val_loss: 0.1471 - val_precision: 0.7102 - val_recall: 0.8976 Epoch 25/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1938 - precision: 0.6089 - recall: 0.9007 - val_loss: 0.1448 - val_precision: 0.7077 - val_recall: 0.8993
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model4, X_test, y_test)
output
156/156 [==============================] - 0s 1ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.970134 | 0.970134 | 0.973761 | 0.971494 |
results.loc[4] = ['1','80','relu',epochs,batch_size,'-','-','-',"SGD_LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
| 4 | 1 | 80 | relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.567311 | 0.193805 | 0.144769 | 0.608861 | 0.707650 | 0.900749 | 0.899306 | 0.973761 | 0.970134 |
Model 5¶
- Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
- SGD with learning rate - 0.01
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - 32
- Class weights taken into account
- Output layer - Sigmoid Activaion Function
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model5 = Sequential()
model5.add(Dense(80,activation="relu",input_dim=X_train.shape[1]))
model5.add(Dense(160,activation="relu"))
model5.add(Dense(1,activation="sigmoid"))
model5.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 80) 3280
dense_1 (Dense) (None, 160) 12960
dense_2 (Dense) (None, 1) 161
=================================================================
Total params: 16401 (64.07 KB)
Trainable params: 16401 (64.07 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
#Add the necessary configurations and fit the model
optimizer = keras.optimizers.SGD(learning_rate=0.01)
model5.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model5.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()
Epoch 1/25 312/312 [==============================] - 3s 6ms/step - loss: 0.4797 - precision: 0.2748 - recall: 0.8523 - val_loss: 0.4100 - val_precision: 0.2544 - val_recall: 0.8715 Epoch 2/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3524 - precision: 0.2733 - recall: 0.8633 - val_loss: 0.3081 - val_precision: 0.3529 - val_recall: 0.8854 Epoch 3/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3108 - precision: 0.3436 - recall: 0.8801 - val_loss: 0.2498 - val_precision: 0.4376 - val_recall: 0.8941 Epoch 4/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2837 - precision: 0.4012 - recall: 0.8858 - val_loss: 0.2195 - val_precision: 0.5010 - val_recall: 0.8906 Epoch 5/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2648 - precision: 0.4500 - recall: 0.8839 - val_loss: 0.2139 - val_precision: 0.5321 - val_recall: 0.9062 Epoch 6/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2475 - precision: 0.4857 - recall: 0.8933 - val_loss: 0.2074 - val_precision: 0.5617 - val_recall: 0.9010 Epoch 7/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2348 - precision: 0.5175 - recall: 0.8876 - val_loss: 0.1834 - val_precision: 0.6061 - val_recall: 0.9028 Epoch 8/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2229 - precision: 0.5711 - recall: 0.8951 - val_loss: 0.2391 - val_precision: 0.4709 - val_recall: 0.9132 Epoch 9/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2131 - precision: 0.5772 - recall: 0.9026 - val_loss: 0.1525 - val_precision: 0.7467 - val_recall: 0.8854 Epoch 10/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2063 - precision: 0.6299 - recall: 0.8989 - val_loss: 0.1570 - val_precision: 0.6803 - val_recall: 0.8976 Epoch 11/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1990 - precision: 0.6343 - recall: 0.8933 - val_loss: 0.1486 - val_precision: 0.7269 - val_recall: 0.8872 Epoch 12/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1922 - precision: 0.6639 - recall: 0.8951 - val_loss: 0.1518 - val_precision: 0.7377 - val_recall: 0.8837 Epoch 13/25 312/312 [==============================] - 1s 5ms/step - loss: 0.1861 - precision: 0.6751 - recall: 0.8989 - val_loss: 0.1591 - val_precision: 0.7007 - val_recall: 0.8941 Epoch 14/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1809 - precision: 0.6823 - recall: 0.8970 - val_loss: 0.1760 - val_precision: 0.6157 - val_recall: 0.9010 Epoch 15/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1740 - precision: 0.6751 - recall: 0.9026 - val_loss: 0.1258 - val_precision: 0.7731 - val_recall: 0.8872 Epoch 16/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1695 - precision: 0.7022 - recall: 0.9007 - val_loss: 0.1241 - val_precision: 0.7703 - val_recall: 0.8906 Epoch 17/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1687 - precision: 0.7074 - recall: 0.9007 - val_loss: 0.1689 - val_precision: 0.6356 - val_recall: 0.8993 Epoch 18/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1622 - precision: 0.7198 - recall: 0.9045 - val_loss: 0.2079 - val_precision: 0.5246 - val_recall: 0.9080 Epoch 19/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1575 - precision: 0.7101 - recall: 0.9082 - val_loss: 0.1897 - val_precision: 0.5471 - val_recall: 0.9080 Epoch 20/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1564 - precision: 0.6931 - recall: 0.9007 - val_loss: 0.1271 - val_precision: 0.7633 - val_recall: 0.8958 Epoch 21/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1529 - precision: 0.7192 - recall: 0.9064 - val_loss: 0.1332 - val_precision: 0.7344 - val_recall: 0.8976 Epoch 22/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1485 - precision: 0.7162 - recall: 0.9026 - val_loss: 0.1360 - val_precision: 0.7161 - val_recall: 0.8976 Epoch 23/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1461 - precision: 0.7132 - recall: 0.9082 - val_loss: 0.1235 - val_precision: 0.7475 - val_recall: 0.8941 Epoch 24/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1403 - precision: 0.7417 - recall: 0.9195 - val_loss: 0.1136 - val_precision: 0.7795 - val_recall: 0.8958 Epoch 25/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1396 - precision: 0.7141 - recall: 0.9120 - val_loss: 0.1157 - val_precision: 0.7674 - val_recall: 0.8993
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model5, X_test, y_test)
output
156/156 [==============================] - 0s 2ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.977551 | 0.977551 | 0.97848 | 0.977946 |
results.loc[5] = ['2','80,160','relu,relu',epochs,batch_size,'-','-','-',"SGD_LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
| 4 | 1 | 80 | relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.567311 | 0.193805 | 0.144769 | 0.608861 | 0.707650 | 0.900749 | 0.899306 | 0.973761 | 0.970134 |
| 5 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.894305 | 0.139635 | 0.115674 | 0.714076 | 0.767407 | 0.911985 | 0.899306 | 0.978480 | 0.977551 |
Model 6¶
- Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
- Adam Optimizer - learning_rate=0.01
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - 32
- Class weights taken into account
- Output layer - Sigmoid Activaion Function
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model6 = Sequential()
model6.add(Dense(80,activation="relu",input_dim=X_train.shape[1]))
model6.add(Dense(160,activation="relu"))
model6.add(Dense(1,activation="sigmoid"))
model6.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 80) 3280
dense_1 (Dense) (None, 160) 12960
dense_2 (Dense) (None, 1) 161
=================================================================
Total params: 16401 (64.07 KB)
Trainable params: 16401 (64.07 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
#Add the necessary configurations and fit the model
adam_optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
model6.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model6.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()
Epoch 1/25 312/312 [==============================] - 2s 5ms/step - loss: 0.4154 - precision: 0.4503 - recall: 0.8685 - val_loss: 0.3027 - val_precision: 0.4393 - val_recall: 0.8854 Epoch 2/25 312/312 [==============================] - 1s 3ms/step - loss: 0.3012 - precision: 0.5477 - recall: 0.8708 - val_loss: 0.2881 - val_precision: 0.5471 - val_recall: 0.8767 Epoch 3/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2393 - precision: 0.6182 - recall: 0.8764 - val_loss: 0.0978 - val_precision: 0.9396 - val_recall: 0.8646 Epoch 4/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2321 - precision: 0.7253 - recall: 0.8951 - val_loss: 0.0745 - val_precision: 0.9221 - val_recall: 0.8837 Epoch 5/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2299 - precision: 0.6547 - recall: 0.8914 - val_loss: 0.1423 - val_precision: 0.8907 - val_recall: 0.8628 Epoch 6/25 312/312 [==============================] - 1s 3ms/step - loss: 0.2875 - precision: 0.5972 - recall: 0.8745 - val_loss: 0.1715 - val_precision: 0.8491 - val_recall: 0.8594 Epoch 7/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2402 - precision: 0.6955 - recall: 0.8895 - val_loss: 0.1320 - val_precision: 0.8545 - val_recall: 0.8767 Epoch 8/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2696 - precision: 0.6389 - recall: 0.8914 - val_loss: 0.3039 - val_precision: 0.4856 - val_recall: 0.9097 Epoch 9/25 312/312 [==============================] - 2s 5ms/step - loss: 0.1807 - precision: 0.7642 - recall: 0.9045 - val_loss: 0.1232 - val_precision: 0.8946 - val_recall: 0.8837 Epoch 10/25 312/312 [==============================] - 2s 5ms/step - loss: 0.1820 - precision: 0.7606 - recall: 0.9045 - val_loss: 0.1354 - val_precision: 0.8304 - val_recall: 0.8924 Epoch 11/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1609 - precision: 0.8036 - recall: 0.9120 - val_loss: 0.1040 - val_precision: 0.9292 - val_recall: 0.8889 Epoch 12/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1989 - precision: 0.7266 - recall: 0.9007 - val_loss: 0.1279 - val_precision: 0.8690 - val_recall: 0.8524 Epoch 13/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2516 - precision: 0.6403 - recall: 0.8933 - val_loss: 0.1660 - val_precision: 0.7896 - val_recall: 0.8924 Epoch 14/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1696 - precision: 0.8039 - recall: 0.9213 - val_loss: 0.1146 - val_precision: 0.9204 - val_recall: 0.8837 Epoch 15/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1730 - precision: 0.7810 - recall: 0.9082 - val_loss: 0.1406 - val_precision: 0.8955 - val_recall: 0.8628 Epoch 16/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1788 - precision: 0.6957 - recall: 0.9120 - val_loss: 0.1523 - val_precision: 0.7936 - val_recall: 0.9010 Epoch 17/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1597 - precision: 0.7945 - recall: 0.9120 - val_loss: 0.1987 - val_precision: 0.5774 - val_recall: 0.8872 Epoch 18/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1487 - precision: 0.8414 - recall: 0.9139 - val_loss: 0.1553 - val_precision: 0.7648 - val_recall: 0.8976 Epoch 19/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1400 - precision: 0.8584 - recall: 0.9195 - val_loss: 0.1304 - val_precision: 0.9036 - val_recall: 0.8785 Epoch 20/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1509 - precision: 0.8180 - recall: 0.9176 - val_loss: 0.1470 - val_precision: 0.8251 - val_recall: 0.8681 Epoch 21/25 312/312 [==============================] - 2s 5ms/step - loss: 0.1792 - precision: 0.7598 - recall: 0.9064 - val_loss: 0.1607 - val_precision: 0.8298 - val_recall: 0.8889 Epoch 22/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1817 - precision: 0.7465 - recall: 0.9101 - val_loss: 0.1472 - val_precision: 0.8082 - val_recall: 0.8924 Epoch 23/25 312/312 [==============================] - 1s 3ms/step - loss: 0.1742 - precision: 0.7614 - recall: 0.9082 - val_loss: 0.1450 - val_precision: 0.7972 - val_recall: 0.8802 Epoch 24/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1692 - precision: 0.8088 - recall: 0.9270 - val_loss: 0.1375 - val_precision: 0.8055 - val_recall: 0.8698 Epoch 25/25 312/312 [==============================] - 1s 4ms/step - loss: 0.1451 - precision: 0.7850 - recall: 0.9232 - val_loss: 0.1335 - val_precision: 0.8403 - val_recall: 0.8767
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model6, X_test, y_test)
output
156/156 [==============================] - 0s 1ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.98176 | 0.98176 | 0.982125 | 0.981924 |
results.loc[6] = ['2','80,160','relu,relu',epochs,batch_size,'-','-','-',"Adam_LR(0.01)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
| 4 | 1 | 80 | relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.567311 | 0.193805 | 0.144769 | 0.608861 | 0.707650 | 0.900749 | 0.899306 | 0.973761 | 0.970134 |
| 5 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.894305 | 0.139635 | 0.115674 | 0.714076 | 0.767407 | 0.911985 | 0.899306 | 0.978480 | 0.977551 |
| 6 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(0.01) | 42.051322 | 0.145135 | 0.133539 | 0.785032 | 0.840266 | 0.923221 | 0.876736 | 0.982125 | 0.981760 |
Model 7¶
- Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
- Adam Optimizer - learning_rate = 1e-5
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - 32
- Class weights taken into account
- Output layer - Sigmoid Activaion Function
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model7 = Sequential()
model7.add(Dense(80,activation="relu",input_dim=X_train.shape[1]))
model7.add(Dense(160,activation="relu"))
model7.add(Dense(1,activation="sigmoid"))
model7.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 80) 3280
dense_1 (Dense) (None, 160) 12960
dense_2 (Dense) (None, 1) 161
=================================================================
Total params: 16401 (64.07 KB)
Trainable params: 16401 (64.07 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
#Add the necessary configurations and fit the model
adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model7.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model7.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()
Epoch 1/25 312/312 [==============================] - 2s 4ms/step - loss: 0.6800 - precision: 0.1310 - recall: 0.8162 - val_loss: 0.7437 - val_precision: 0.0772 - val_recall: 0.8229 Epoch 2/25 312/312 [==============================] - 1s 3ms/step - loss: 0.6111 - precision: 0.0810 - recall: 0.8596 - val_loss: 0.6941 - val_precision: 0.0943 - val_recall: 0.8733 Epoch 3/25 312/312 [==============================] - 1s 3ms/step - loss: 0.5609 - precision: 0.0976 - recall: 0.8876 - val_loss: 0.6429 - val_precision: 0.1190 - val_recall: 0.8750 Epoch 4/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5231 - precision: 0.1218 - recall: 0.8745 - val_loss: 0.5984 - val_precision: 0.1495 - val_recall: 0.8628 Epoch 5/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4934 - precision: 0.1468 - recall: 0.8689 - val_loss: 0.5612 - val_precision: 0.1705 - val_recall: 0.8594 Epoch 6/25 312/312 [==============================] - 2s 5ms/step - loss: 0.4699 - precision: 0.1650 - recall: 0.8689 - val_loss: 0.5296 - val_precision: 0.1881 - val_recall: 0.8646 Epoch 7/25 312/312 [==============================] - 2s 5ms/step - loss: 0.4510 - precision: 0.1781 - recall: 0.8652 - val_loss: 0.5022 - val_precision: 0.2011 - val_recall: 0.8681 Epoch 8/25 312/312 [==============================] - 1s 3ms/step - loss: 0.4352 - precision: 0.1919 - recall: 0.8633 - val_loss: 0.4808 - val_precision: 0.2103 - val_recall: 0.8698 Epoch 9/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4219 - precision: 0.1984 - recall: 0.8689 - val_loss: 0.4627 - val_precision: 0.2197 - val_recall: 0.8715 Epoch 10/25 312/312 [==============================] - 1s 3ms/step - loss: 0.4105 - precision: 0.2083 - recall: 0.8727 - val_loss: 0.4447 - val_precision: 0.2308 - val_recall: 0.8750 Epoch 11/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4004 - precision: 0.2157 - recall: 0.8727 - val_loss: 0.4285 - val_precision: 0.2388 - val_recall: 0.8715 Epoch 12/25 312/312 [==============================] - 1s 3ms/step - loss: 0.3914 - precision: 0.2230 - recall: 0.8708 - val_loss: 0.4153 - val_precision: 0.2471 - val_recall: 0.8750 Epoch 13/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3834 - precision: 0.2304 - recall: 0.8745 - val_loss: 0.4037 - val_precision: 0.2528 - val_recall: 0.8750 Epoch 14/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3760 - precision: 0.2381 - recall: 0.8745 - val_loss: 0.3921 - val_precision: 0.2597 - val_recall: 0.8802 Epoch 15/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3694 - precision: 0.2420 - recall: 0.8783 - val_loss: 0.3833 - val_precision: 0.2650 - val_recall: 0.8819 Epoch 16/25 312/312 [==============================] - 1s 5ms/step - loss: 0.3631 - precision: 0.2474 - recall: 0.8801 - val_loss: 0.3713 - val_precision: 0.2752 - val_recall: 0.8819 Epoch 17/25 312/312 [==============================] - 2s 5ms/step - loss: 0.3574 - precision: 0.2610 - recall: 0.8783 - val_loss: 0.3633 - val_precision: 0.2828 - val_recall: 0.8872 Epoch 18/25 312/312 [==============================] - 1s 5ms/step - loss: 0.3521 - precision: 0.2669 - recall: 0.8801 - val_loss: 0.3570 - val_precision: 0.2897 - val_recall: 0.8941 Epoch 19/25 312/312 [==============================] - 1s 3ms/step - loss: 0.3472 - precision: 0.2695 - recall: 0.8801 - val_loss: 0.3487 - val_precision: 0.2958 - val_recall: 0.8941 Epoch 20/25 312/312 [==============================] - 1s 3ms/step - loss: 0.3425 - precision: 0.2764 - recall: 0.8820 - val_loss: 0.3440 - val_precision: 0.2998 - val_recall: 0.8993 Epoch 21/25 312/312 [==============================] - 1s 3ms/step - loss: 0.3381 - precision: 0.2818 - recall: 0.8839 - val_loss: 0.3374 - val_precision: 0.3087 - val_recall: 0.9010 Epoch 22/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3340 - precision: 0.2874 - recall: 0.8820 - val_loss: 0.3305 - val_precision: 0.3163 - val_recall: 0.9010 Epoch 23/25 312/312 [==============================] - 1s 3ms/step - loss: 0.3301 - precision: 0.2940 - recall: 0.8820 - val_loss: 0.3250 - val_precision: 0.3220 - val_recall: 0.9010 Epoch 24/25 312/312 [==============================] - 1s 3ms/step - loss: 0.3264 - precision: 0.3002 - recall: 0.8876 - val_loss: 0.3192 - val_precision: 0.3297 - val_recall: 0.8993 Epoch 25/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3230 - precision: 0.3053 - recall: 0.8914 - val_loss: 0.3132 - val_precision: 0.3372 - val_recall: 0.9010
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model7, X_test, y_test)
output
156/156 [==============================] - 0s 1ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.898577 | 0.898577 | 0.953981 | 0.917995 |
results.loc[7] = ['2','80,160','relu,relu',epochs,batch_size,'-','-','-',"Adam_LR(le-5)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
| 4 | 1 | 80 | relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.567311 | 0.193805 | 0.144769 | 0.608861 | 0.707650 | 0.900749 | 0.899306 | 0.973761 | 0.970134 |
| 5 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.894305 | 0.139635 | 0.115674 | 0.714076 | 0.767407 | 0.911985 | 0.899306 | 0.978480 | 0.977551 |
| 6 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(0.01) | 42.051322 | 0.145135 | 0.133539 | 0.785032 | 0.840266 | 0.923221 | 0.876736 | 0.982125 | 0.981760 |
| 7 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(le-5) | 31.487571 | 0.322984 | 0.313230 | 0.305324 | 0.337232 | 0.891386 | 0.901042 | 0.953981 | 0.898577 |
Model 8¶
- Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
- Adam Optimizer - learning_rate = 1e-5
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - 32
- Class weights taken into account
- Output layer - Sigmoid Activaion Function
- Drop out - 0.3
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model8 = Sequential([
Dense(80, activation='relu', input_shape=(X_train.shape[1],)),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(160, activation='relu'),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(1, activation='sigmoid') # For binary classification
])
model8.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 80) 3280
dropout (Dropout) (None, 80) 0
dense_1 (Dense) (None, 160) 12960
dropout_1 (Dropout) (None, 160) 0
dense_2 (Dense) (None, 1) 161
=================================================================
Total params: 16401 (64.07 KB)
Trainable params: 16401 (64.07 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
#Add the necessary configurations and fit the model
adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model8.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model8.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()
Epoch 1/25 312/312 [==============================] - 3s 6ms/step - loss: 0.7853 - precision: 0.1638 - recall: 0.5477 - val_loss: 0.6004 - val_precision: 0.0592 - val_recall: 0.1354 Epoch 2/25 312/312 [==============================] - 2s 5ms/step - loss: 0.7380 - precision: 0.0628 - recall: 0.3165 - val_loss: 0.6149 - val_precision: 0.0877 - val_recall: 0.3264 Epoch 3/25 312/312 [==============================] - 1s 4ms/step - loss: 0.6912 - precision: 0.0761 - recall: 0.4401 - val_loss: 0.6224 - val_precision: 0.1154 - val_recall: 0.5556 Epoch 4/25 312/312 [==============================] - 1s 4ms/step - loss: 0.6641 - precision: 0.0843 - recall: 0.5356 - val_loss: 0.6250 - val_precision: 0.1253 - val_recall: 0.7014 Epoch 5/25 312/312 [==============================] - 1s 4ms/step - loss: 0.6353 - precision: 0.0949 - recall: 0.6236 - val_loss: 0.6221 - val_precision: 0.1339 - val_recall: 0.8038 Epoch 6/25 312/312 [==============================] - 1s 3ms/step - loss: 0.6229 - precision: 0.0948 - recall: 0.6255 - val_loss: 0.6194 - val_precision: 0.1370 - val_recall: 0.8472 Epoch 7/25 312/312 [==============================] - 1s 3ms/step - loss: 0.5947 - precision: 0.1022 - recall: 0.6910 - val_loss: 0.6112 - val_precision: 0.1412 - val_recall: 0.8681 Epoch 8/25 312/312 [==============================] - 1s 3ms/step - loss: 0.5726 - precision: 0.1145 - recall: 0.7528 - val_loss: 0.6006 - val_precision: 0.1460 - val_recall: 0.8750 Epoch 9/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5600 - precision: 0.1207 - recall: 0.7865 - val_loss: 0.5916 - val_precision: 0.1503 - val_recall: 0.8872 Epoch 10/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5448 - precision: 0.1264 - recall: 0.7978 - val_loss: 0.5800 - val_precision: 0.1546 - val_recall: 0.8889 Epoch 11/25 312/312 [==============================] - 2s 5ms/step - loss: 0.5312 - precision: 0.1314 - recall: 0.8221 - val_loss: 0.5673 - val_precision: 0.1600 - val_recall: 0.8906 Epoch 12/25 312/312 [==============================] - 2s 5ms/step - loss: 0.5257 - precision: 0.1360 - recall: 0.8240 - val_loss: 0.5579 - val_precision: 0.1631 - val_recall: 0.8924 Epoch 13/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5132 - precision: 0.1374 - recall: 0.8090 - val_loss: 0.5472 - val_precision: 0.1667 - val_recall: 0.8924 Epoch 14/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4953 - precision: 0.1465 - recall: 0.8296 - val_loss: 0.5347 - val_precision: 0.1721 - val_recall: 0.8941 Epoch 15/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4891 - precision: 0.1461 - recall: 0.8221 - val_loss: 0.5224 - val_precision: 0.1773 - val_recall: 0.8889 Epoch 16/25 312/312 [==============================] - 1s 3ms/step - loss: 0.4845 - precision: 0.1559 - recall: 0.8315 - val_loss: 0.5104 - val_precision: 0.1825 - val_recall: 0.8889 Epoch 17/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4729 - precision: 0.1630 - recall: 0.8352 - val_loss: 0.4988 - val_precision: 0.1875 - val_recall: 0.8872 Epoch 18/25 312/312 [==============================] - 1s 3ms/step - loss: 0.4729 - precision: 0.1639 - recall: 0.8333 - val_loss: 0.4930 - val_precision: 0.1894 - val_recall: 0.8889 Epoch 19/25 312/312 [==============================] - 1s 3ms/step - loss: 0.4564 - precision: 0.1716 - recall: 0.8483 - val_loss: 0.4839 - val_precision: 0.1944 - val_recall: 0.8906 Epoch 20/25 312/312 [==============================] - 1s 3ms/step - loss: 0.4639 - precision: 0.1694 - recall: 0.8446 - val_loss: 0.4789 - val_precision: 0.1962 - val_recall: 0.8941 Epoch 21/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4497 - precision: 0.1729 - recall: 0.8427 - val_loss: 0.4713 - val_precision: 0.2007 - val_recall: 0.8941 Epoch 22/25 312/312 [==============================] - 2s 5ms/step - loss: 0.4394 - precision: 0.1810 - recall: 0.8652 - val_loss: 0.4619 - val_precision: 0.2062 - val_recall: 0.8941 Epoch 23/25 312/312 [==============================] - 2s 5ms/step - loss: 0.4362 - precision: 0.1814 - recall: 0.8521 - val_loss: 0.4536 - val_precision: 0.2112 - val_recall: 0.8958 Epoch 24/25 312/312 [==============================] - 1s 3ms/step - loss: 0.4406 - precision: 0.1877 - recall: 0.8521 - val_loss: 0.4470 - val_precision: 0.2147 - val_recall: 0.8958 Epoch 25/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4252 - precision: 0.1880 - recall: 0.8483 - val_loss: 0.4401 - val_precision: 0.2189 - val_recall: 0.8958
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model8, X_test, y_test)
output
156/156 [==============================] - 0s 2ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.818 | 0.818 | 0.946604 | 0.863412 |
results.loc[8] = ['2','80,160','relu,relu',epochs,batch_size,'-','-','0.3',"Adam_LR(le-5)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
| 4 | 1 | 80 | relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.567311 | 0.193805 | 0.144769 | 0.608861 | 0.707650 | 0.900749 | 0.899306 | 0.973761 | 0.970134 |
| 5 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.894305 | 0.139635 | 0.115674 | 0.714076 | 0.767407 | 0.911985 | 0.899306 | 0.978480 | 0.977551 |
| 6 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(0.01) | 42.051322 | 0.145135 | 0.133539 | 0.785032 | 0.840266 | 0.923221 | 0.876736 | 0.982125 | 0.981760 |
| 7 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(le-5) | 31.487571 | 0.322984 | 0.313230 | 0.305324 | 0.337232 | 0.891386 | 0.901042 | 0.953981 | 0.898577 |
| 8 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | 0.3 | Adam_LR(le-5) | 42.194058 | 0.425188 | 0.440071 | 0.187967 | 0.218922 | 0.848315 | 0.895833 | 0.946604 | 0.818000 |
Model 9¶
- Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
- Adam Optimizer - learning_rate = 1e-5
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - 32
- Class weights taken into account
- Output layer - Sigmoid Activaion Function
- Drop out - 0.3
- Batch Normalization added
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model9 = Sequential([
Dense(80, activation='relu', input_shape=(X_train.shape[1],)),
BatchNormalization(),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(160, activation='relu'),
BatchNormalization(),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(1, activation='sigmoid') # For binary classification
])
model9.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 80) 3280
batch_normalization (Batch (None, 80) 320
Normalization)
dropout (Dropout) (None, 80) 0
dense_1 (Dense) (None, 160) 12960
batch_normalization_1 (Bat (None, 160) 640
chNormalization)
dropout_1 (Dropout) (None, 160) 0
dense_2 (Dense) (None, 1) 161
=================================================================
Total params: 17361 (67.82 KB)
Trainable params: 16881 (65.94 KB)
Non-trainable params: 480 (1.88 KB)
_________________________________________________________________
#Add the necessary configurations and fit the model
adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model9.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model9.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()
Epoch 1/25 312/312 [==============================] - 3s 5ms/step - loss: 0.9218 - precision: 0.1141 - recall: 0.7595 - val_loss: 0.7801 - val_precision: 0.0820 - val_recall: 0.7031 Epoch 2/25 312/312 [==============================] - 2s 5ms/step - loss: 0.8392 - precision: 0.0732 - recall: 0.6873 - val_loss: 0.7897 - val_precision: 0.0940 - val_recall: 0.7847 Epoch 3/25 312/312 [==============================] - 2s 6ms/step - loss: 0.7696 - precision: 0.0799 - recall: 0.7322 - val_loss: 0.7539 - val_precision: 0.1054 - val_recall: 0.8281 Epoch 4/25 312/312 [==============================] - 3s 9ms/step - loss: 0.7185 - precision: 0.0837 - recall: 0.7715 - val_loss: 0.7420 - val_precision: 0.1118 - val_recall: 0.8663 Epoch 5/25 312/312 [==============================] - 2s 7ms/step - loss: 0.7302 - precision: 0.0850 - recall: 0.7509 - val_loss: 0.7191 - val_precision: 0.1195 - val_recall: 0.8819 Epoch 6/25 312/312 [==============================] - 2s 6ms/step - loss: 0.6561 - precision: 0.0945 - recall: 0.8146 - val_loss: 0.6921 - val_precision: 0.1264 - val_recall: 0.8958 Epoch 7/25 312/312 [==============================] - 2s 5ms/step - loss: 0.6192 - precision: 0.0976 - recall: 0.8390 - val_loss: 0.6704 - val_precision: 0.1338 - val_recall: 0.9045 Epoch 8/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5946 - precision: 0.1013 - recall: 0.8464 - val_loss: 0.6491 - val_precision: 0.1392 - val_recall: 0.9097 Epoch 9/25 312/312 [==============================] - 2s 7ms/step - loss: 0.6136 - precision: 0.1017 - recall: 0.8240 - val_loss: 0.6278 - val_precision: 0.1455 - val_recall: 0.9132 Epoch 10/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5825 - precision: 0.1052 - recall: 0.8483 - val_loss: 0.6016 - val_precision: 0.1556 - val_recall: 0.9184 Epoch 11/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5695 - precision: 0.1076 - recall: 0.8502 - val_loss: 0.5851 - val_precision: 0.1623 - val_recall: 0.9219 Epoch 12/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5742 - precision: 0.1089 - recall: 0.8390 - val_loss: 0.5732 - val_precision: 0.1677 - val_recall: 0.9236 Epoch 13/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5496 - precision: 0.1123 - recall: 0.8502 - val_loss: 0.5427 - val_precision: 0.1811 - val_recall: 0.9201 Epoch 14/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5355 - precision: 0.1197 - recall: 0.8745 - val_loss: 0.5373 - val_precision: 0.1849 - val_recall: 0.9201 Epoch 15/25 312/312 [==============================] - 2s 5ms/step - loss: 0.5240 - precision: 0.1200 - recall: 0.8764 - val_loss: 0.5236 - val_precision: 0.1922 - val_recall: 0.9201 Epoch 16/25 312/312 [==============================] - 2s 5ms/step - loss: 0.5240 - precision: 0.1203 - recall: 0.8483 - val_loss: 0.5100 - val_precision: 0.2000 - val_recall: 0.9236 Epoch 17/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5101 - precision: 0.1240 - recall: 0.8558 - val_loss: 0.4974 - val_precision: 0.2071 - val_recall: 0.9236 Epoch 18/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4940 - precision: 0.1289 - recall: 0.8652 - val_loss: 0.4796 - val_precision: 0.2161 - val_recall: 0.9219 Epoch 19/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4877 - precision: 0.1302 - recall: 0.8596 - val_loss: 0.4717 - val_precision: 0.2232 - val_recall: 0.9236 Epoch 20/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4737 - precision: 0.1364 - recall: 0.8839 - val_loss: 0.4522 - val_precision: 0.2362 - val_recall: 0.9253 Epoch 21/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4730 - precision: 0.1389 - recall: 0.8614 - val_loss: 0.4400 - val_precision: 0.2454 - val_recall: 0.9253 Epoch 22/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4645 - precision: 0.1423 - recall: 0.8783 - val_loss: 0.4435 - val_precision: 0.2437 - val_recall: 0.9271 Epoch 23/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4566 - precision: 0.1451 - recall: 0.8839 - val_loss: 0.4228 - val_precision: 0.2573 - val_recall: 0.9219 Epoch 24/25 312/312 [==============================] - 1s 5ms/step - loss: 0.4600 - precision: 0.1430 - recall: 0.8408 - val_loss: 0.4124 - val_precision: 0.2642 - val_recall: 0.9219 Epoch 25/25 312/312 [==============================] - 2s 6ms/step - loss: 0.4508 - precision: 0.1456 - recall: 0.8670 - val_loss: 0.4017 - val_precision: 0.2709 - val_recall: 0.9236
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model9, X_test, y_test)
output
156/156 [==============================] - 0s 1ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.854279 | 0.854279 | 0.951479 | 0.888202 |
results.loc[9] = ['2','80,160','relu,relu',epochs,batch_size,'yes','-','0.3',"Adam_LR(le-5)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
| 4 | 1 | 80 | relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.567311 | 0.193805 | 0.144769 | 0.608861 | 0.707650 | 0.900749 | 0.899306 | 0.973761 | 0.970134 |
| 5 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.894305 | 0.139635 | 0.115674 | 0.714076 | 0.767407 | 0.911985 | 0.899306 | 0.978480 | 0.977551 |
| 6 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(0.01) | 42.051322 | 0.145135 | 0.133539 | 0.785032 | 0.840266 | 0.923221 | 0.876736 | 0.982125 | 0.981760 |
| 7 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(le-5) | 31.487571 | 0.322984 | 0.313230 | 0.305324 | 0.337232 | 0.891386 | 0.901042 | 0.953981 | 0.898577 |
| 8 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | 0.3 | Adam_LR(le-5) | 42.194058 | 0.425188 | 0.440071 | 0.187967 | 0.218922 | 0.848315 | 0.895833 | 0.946604 | 0.818000 |
| 9 | 2 | 80,160 | relu,relu | 25 | 32 | yes | - | 0.3 | Adam_LR(le-5) | 42.708833 | 0.450809 | 0.401692 | 0.145643 | 0.270876 | 0.867041 | 0.923611 | 0.951479 | 0.854279 |
Model 10¶
- Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
- Adam Optimizer - learning_rate = 1e-5
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - 32
- Class weights taken into account
- Output layer - Sigmoid Activaion Function
- Drop out - 0.3
- Batch Normalization added
- Weight Initialization done - He (HL1), Glorot (HL2), RandomNormal (Output)
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model10 = Sequential([
Dense(80, activation='relu', kernel_initializer=HeNormal(), input_shape=(X_train.shape[1],)),
BatchNormalization(),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(160, activation='relu', kernel_initializer=GlorotUniform()),
BatchNormalization(),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(1, activation='sigmoid', kernel_initializer=RandomNormal()) # For binary classification
])
model10.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 80) 3280
batch_normalization (Batch (None, 80) 320
Normalization)
dropout (Dropout) (None, 80) 0
dense_1 (Dense) (None, 160) 12960
batch_normalization_1 (Bat (None, 160) 640
chNormalization)
dropout_1 (Dropout) (None, 160) 0
dense_2 (Dense) (None, 1) 161
=================================================================
Total params: 17361 (67.82 KB)
Trainable params: 16881 (65.94 KB)
Non-trainable params: 480 (1.88 KB)
_________________________________________________________________
#Add the necessary configurations and fit the model
adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model10.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model10.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()
Epoch 1/25 312/312 [==============================] - 3s 5ms/step - loss: 0.7269 - precision: 0.1210 - recall: 0.7658 - val_loss: 0.7111 - val_precision: 0.0803 - val_recall: 0.7170 Epoch 2/25 312/312 [==============================] - 1s 4ms/step - loss: 0.6642 - precision: 0.0736 - recall: 0.6948 - val_loss: 0.7010 - val_precision: 0.0942 - val_recall: 0.8507 Epoch 3/25 312/312 [==============================] - 1s 5ms/step - loss: 0.6309 - precision: 0.0831 - recall: 0.7865 - val_loss: 0.6854 - val_precision: 0.1016 - val_recall: 0.8906 Epoch 4/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5977 - precision: 0.0886 - recall: 0.8221 - val_loss: 0.6756 - val_precision: 0.1084 - val_recall: 0.9045 Epoch 5/25 312/312 [==============================] - 2s 5ms/step - loss: 0.5854 - precision: 0.0906 - recall: 0.8258 - val_loss: 0.6626 - val_precision: 0.1170 - val_recall: 0.9184 Epoch 6/25 312/312 [==============================] - 2s 6ms/step - loss: 0.5702 - precision: 0.0942 - recall: 0.8390 - val_loss: 0.6531 - val_precision: 0.1230 - val_recall: 0.9253 Epoch 7/25 312/312 [==============================] - 2s 5ms/step - loss: 0.5496 - precision: 0.0971 - recall: 0.8577 - val_loss: 0.6412 - val_precision: 0.1297 - val_recall: 0.9253 Epoch 8/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5409 - precision: 0.1035 - recall: 0.8820 - val_loss: 0.6338 - val_precision: 0.1343 - val_recall: 0.9271 Epoch 9/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5313 - precision: 0.1037 - recall: 0.8708 - val_loss: 0.6244 - val_precision: 0.1398 - val_recall: 0.9288 Epoch 10/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5142 - precision: 0.1094 - recall: 0.8820 - val_loss: 0.6149 - val_precision: 0.1449 - val_recall: 0.9288 Epoch 11/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5080 - precision: 0.1133 - recall: 0.8951 - val_loss: 0.6114 - val_precision: 0.1478 - val_recall: 0.9288 Epoch 12/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4982 - precision: 0.1132 - recall: 0.8839 - val_loss: 0.6052 - val_precision: 0.1520 - val_recall: 0.9358 Epoch 13/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4979 - precision: 0.1177 - recall: 0.8839 - val_loss: 0.5900 - val_precision: 0.1604 - val_recall: 0.9340 Epoch 14/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4873 - precision: 0.1194 - recall: 0.8914 - val_loss: 0.5873 - val_precision: 0.1616 - val_recall: 0.9358 Epoch 15/25 312/312 [==============================] - 2s 5ms/step - loss: 0.4827 - precision: 0.1211 - recall: 0.8989 - val_loss: 0.5790 - val_precision: 0.1662 - val_recall: 0.9340 Epoch 16/25 312/312 [==============================] - 2s 8ms/step - loss: 0.4811 - precision: 0.1223 - recall: 0.8989 - val_loss: 0.5698 - val_precision: 0.1699 - val_recall: 0.9340 Epoch 17/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4690 - precision: 0.1251 - recall: 0.8933 - val_loss: 0.5629 - val_precision: 0.1747 - val_recall: 0.9340 Epoch 18/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4673 - precision: 0.1274 - recall: 0.8858 - val_loss: 0.5494 - val_precision: 0.1811 - val_recall: 0.9323 Epoch 19/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4571 - precision: 0.1324 - recall: 0.8914 - val_loss: 0.5408 - val_precision: 0.1861 - val_recall: 0.9323 Epoch 20/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4491 - precision: 0.1333 - recall: 0.9007 - val_loss: 0.5309 - val_precision: 0.1925 - val_recall: 0.9306 Epoch 21/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4522 - precision: 0.1341 - recall: 0.8933 - val_loss: 0.5210 - val_precision: 0.1993 - val_recall: 0.9323 Epoch 22/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4425 - precision: 0.1388 - recall: 0.8989 - val_loss: 0.5222 - val_precision: 0.1987 - val_recall: 0.9323 Epoch 23/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4430 - precision: 0.1403 - recall: 0.9045 - val_loss: 0.5150 - val_precision: 0.2032 - val_recall: 0.9306 Epoch 24/25 312/312 [==============================] - 1s 5ms/step - loss: 0.4377 - precision: 0.1420 - recall: 0.8858 - val_loss: 0.5036 - val_precision: 0.2111 - val_recall: 0.9288 Epoch 25/25 312/312 [==============================] - 2s 6ms/step - loss: 0.4369 - precision: 0.1449 - recall: 0.8951 - val_loss: 0.4944 - val_precision: 0.2176 - val_recall: 0.9271
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model10, X_test, y_test)
output
156/156 [==============================] - 0s 2ms/step
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.801764 | 0.801764 | 0.948019 | 0.852611 |
results.loc[10] = ['2','80,160','relu,relu',epochs,batch_size,'yes','he/glorot/randnorm','0.3',"Adam_LR(le-5)",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
| 4 | 1 | 80 | relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.567311 | 0.193805 | 0.144769 | 0.608861 | 0.707650 | 0.900749 | 0.899306 | 0.973761 | 0.970134 |
| 5 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.894305 | 0.139635 | 0.115674 | 0.714076 | 0.767407 | 0.911985 | 0.899306 | 0.978480 | 0.977551 |
| 6 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(0.01) | 42.051322 | 0.145135 | 0.133539 | 0.785032 | 0.840266 | 0.923221 | 0.876736 | 0.982125 | 0.981760 |
| 7 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(le-5) | 31.487571 | 0.322984 | 0.313230 | 0.305324 | 0.337232 | 0.891386 | 0.901042 | 0.953981 | 0.898577 |
| 8 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | 0.3 | Adam_LR(le-5) | 42.194058 | 0.425188 | 0.440071 | 0.187967 | 0.218922 | 0.848315 | 0.895833 | 0.946604 | 0.818000 |
| 9 | 2 | 80,160 | relu,relu | 25 | 32 | yes | - | 0.3 | Adam_LR(le-5) | 42.708833 | 0.450809 | 0.401692 | 0.145643 | 0.270876 | 0.867041 | 0.923611 | 0.951479 | 0.854279 |
| 10 | 2 | 80,160 | relu,relu | 25 | 32 | yes | he/glorot/randnorm | 0.3 | Adam_LR(le-5) | 37.227174 | 0.436856 | 0.494443 | 0.144936 | 0.217604 | 0.895131 | 0.927083 | 0.948019 | 0.801764 |
Model 11¶
- Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
- Adam Optimizer - learning_rate = 1e-5
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - 32
- Class weights taken into account
- Output layer - Sigmoid Activaion Function
- Drop out - 0.3
- Batch Normalization added
- Weight Initialization done - He (HL1), Glorot (HL2), RandomNormal (Output)
- Regularization - L1 and L2
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model11 = Sequential([
Dense(80, activation='relu', kernel_initializer=HeNormal(),kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4), input_shape=(X_train.shape[1],)),
BatchNormalization(),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(160, activation='relu', kernel_initializer=GlorotUniform(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)),
BatchNormalization(),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(1, activation='sigmoid', kernel_initializer=RandomNormal(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)) # For binary classification
])
model11.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 80) 3280
batch_normalization (Batch (None, 80) 320
Normalization)
dropout (Dropout) (None, 80) 0
dense_1 (Dense) (None, 160) 12960
batch_normalization_1 (Bat (None, 160) 640
chNormalization)
dropout_1 (Dropout) (None, 160) 0
dense_2 (Dense) (None, 1) 161
=================================================================
Total params: 17361 (67.82 KB)
Trainable params: 16881 (65.94 KB)
Non-trainable params: 480 (1.88 KB)
_________________________________________________________________
#Add the necessary configurations and fit the model
adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model11.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model11.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()
Epoch 1/25 312/312 [==============================] - 3s 6ms/step - loss: 0.7786 - precision: 0.1103 - recall: 0.7459 - val_loss: 0.7860 - val_precision: 0.0763 - val_recall: 0.7517 Epoch 2/25 312/312 [==============================] - 1s 4ms/step - loss: 0.7263 - precision: 0.0701 - recall: 0.6573 - val_loss: 0.7735 - val_precision: 0.0867 - val_recall: 0.8264 Epoch 3/25 312/312 [==============================] - 1s 5ms/step - loss: 0.6839 - precision: 0.0802 - recall: 0.7472 - val_loss: 0.7607 - val_precision: 0.0961 - val_recall: 0.8698 Epoch 4/25 312/312 [==============================] - 1s 4ms/step - loss: 0.6414 - precision: 0.0883 - recall: 0.7921 - val_loss: 0.7482 - val_precision: 0.1061 - val_recall: 0.8924 Epoch 5/25 312/312 [==============================] - 1s 5ms/step - loss: 0.6184 - precision: 0.0943 - recall: 0.8333 - val_loss: 0.7419 - val_precision: 0.1114 - val_recall: 0.9010 Epoch 6/25 312/312 [==============================] - 2s 7ms/step - loss: 0.5992 - precision: 0.1005 - recall: 0.8539 - val_loss: 0.7221 - val_precision: 0.1216 - val_recall: 0.9062 Epoch 7/25 312/312 [==============================] - 2s 6ms/step - loss: 0.5878 - precision: 0.1032 - recall: 0.8633 - val_loss: 0.7159 - val_precision: 0.1261 - val_recall: 0.9149 Epoch 8/25 312/312 [==============================] - 1s 5ms/step - loss: 0.5771 - precision: 0.1054 - recall: 0.8745 - val_loss: 0.7099 - val_precision: 0.1309 - val_recall: 0.9219 Epoch 9/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5561 - precision: 0.1115 - recall: 0.8895 - val_loss: 0.6992 - val_precision: 0.1378 - val_recall: 0.9253 Epoch 10/25 312/312 [==============================] - 1s 5ms/step - loss: 0.5564 - precision: 0.1116 - recall: 0.8801 - val_loss: 0.6869 - val_precision: 0.1439 - val_recall: 0.9271 Epoch 11/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5404 - precision: 0.1182 - recall: 0.8895 - val_loss: 0.6824 - val_precision: 0.1471 - val_recall: 0.9306 Epoch 12/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5326 - precision: 0.1176 - recall: 0.8764 - val_loss: 0.6763 - val_precision: 0.1504 - val_recall: 0.9340 Epoch 13/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5320 - precision: 0.1208 - recall: 0.8839 - val_loss: 0.6621 - val_precision: 0.1568 - val_recall: 0.9288 Epoch 14/25 312/312 [==============================] - 1s 5ms/step - loss: 0.5313 - precision: 0.1228 - recall: 0.8708 - val_loss: 0.6530 - val_precision: 0.1604 - val_recall: 0.9253 Epoch 15/25 312/312 [==============================] - 2s 5ms/step - loss: 0.5145 - precision: 0.1280 - recall: 0.9007 - val_loss: 0.6541 - val_precision: 0.1610 - val_recall: 0.9306 Epoch 16/25 312/312 [==============================] - 2s 6ms/step - loss: 0.5112 - precision: 0.1297 - recall: 0.8951 - val_loss: 0.6406 - val_precision: 0.1670 - val_recall: 0.9288 Epoch 17/25 312/312 [==============================] - 2s 5ms/step - loss: 0.5002 - precision: 0.1325 - recall: 0.9026 - val_loss: 0.6330 - val_precision: 0.1714 - val_recall: 0.9288 Epoch 18/25 312/312 [==============================] - 1s 4ms/step - loss: 0.5064 - precision: 0.1324 - recall: 0.8895 - val_loss: 0.6191 - val_precision: 0.1778 - val_recall: 0.9288 Epoch 19/25 312/312 [==============================] - 1s 5ms/step - loss: 0.4869 - precision: 0.1389 - recall: 0.9139 - val_loss: 0.6147 - val_precision: 0.1801 - val_recall: 0.9306 Epoch 20/25 312/312 [==============================] - 1s 5ms/step - loss: 0.4901 - precision: 0.1385 - recall: 0.8989 - val_loss: 0.5995 - val_precision: 0.1867 - val_recall: 0.9271 Epoch 21/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4845 - precision: 0.1403 - recall: 0.8876 - val_loss: 0.5976 - val_precision: 0.1897 - val_recall: 0.9306 Epoch 22/25 312/312 [==============================] - 1s 5ms/step - loss: 0.4734 - precision: 0.1475 - recall: 0.9082 - val_loss: 0.5916 - val_precision: 0.1939 - val_recall: 0.9306 Epoch 23/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4810 - precision: 0.1494 - recall: 0.9007 - val_loss: 0.5787 - val_precision: 0.2008 - val_recall: 0.9288 Epoch 24/25 312/312 [==============================] - 1s 5ms/step - loss: 0.4694 - precision: 0.1490 - recall: 0.8951 - val_loss: 0.5682 - val_precision: 0.2070 - val_recall: 0.9253 Epoch 25/25 312/312 [==============================] - 2s 6ms/step - loss: 0.4675 - precision: 0.1506 - recall: 0.8951 - val_loss: 0.5597 - val_precision: 0.2135 - val_recall: 0.9253
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model11, X_test, y_test)
print(output)
output_1 = model_performance(model11, X_train, y_train)
print(output_1)
156/156 [==============================] - 0s 1ms/step Accuracy Recall Precision F1 Score 0 0.797354 0.797354 0.946944 0.849524 312/312 [==============================] - 0s 1ms/step Accuracy Recall Precision F1 Score 0 0.794931 0.794931 0.951182 0.849408
results.loc[11] = ['2','80,160','relu,relu',epochs,batch_size,'yes','he/glorot/randnorm','0.3',"Adam_LR(le-5)/L1,L2",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
| 4 | 1 | 80 | relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.567311 | 0.193805 | 0.144769 | 0.608861 | 0.707650 | 0.900749 | 0.899306 | 0.973761 | 0.970134 |
| 5 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.894305 | 0.139635 | 0.115674 | 0.714076 | 0.767407 | 0.911985 | 0.899306 | 0.978480 | 0.977551 |
| 6 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(0.01) | 42.051322 | 0.145135 | 0.133539 | 0.785032 | 0.840266 | 0.923221 | 0.876736 | 0.982125 | 0.981760 |
| 7 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(le-5) | 31.487571 | 0.322984 | 0.313230 | 0.305324 | 0.337232 | 0.891386 | 0.901042 | 0.953981 | 0.898577 |
| 8 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | 0.3 | Adam_LR(le-5) | 42.194058 | 0.425188 | 0.440071 | 0.187967 | 0.218922 | 0.848315 | 0.895833 | 0.946604 | 0.818000 |
| 9 | 2 | 80,160 | relu,relu | 25 | 32 | yes | - | 0.3 | Adam_LR(le-5) | 42.708833 | 0.450809 | 0.401692 | 0.145643 | 0.270876 | 0.867041 | 0.923611 | 0.951479 | 0.854279 |
| 10 | 2 | 80,160 | relu,relu | 25 | 32 | yes | he/glorot/randnorm | 0.3 | Adam_LR(le-5) | 37.227174 | 0.436856 | 0.494443 | 0.144936 | 0.217604 | 0.895131 | 0.927083 | 0.948019 | 0.801764 |
| 11 | 2 | 80,160 | relu,relu | 25 | 32 | yes | he/glorot/randnorm | 0.3 | Adam_LR(le-5)/L1,L2 | 42.635221 | 0.467466 | 0.559652 | 0.150646 | 0.213456 | 0.895131 | 0.925347 | 0.946944 | 0.797354 |
Model 12¶
- Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
- AdaGrad Optimizer - learning_rate = 0.01
- Loss function - Binary Cross entropy
- Epochs - 25
- Batch size - 32
- Class weights taken into account
- Output layer - Sigmoid Activaion Function
- Drop out - 0.3
- Batch Normalization added
- Weight Initialization done - He (HL1), Glorot (HL2), RandomNormal (Output)
- Regularization - L1 and L2
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model12 = Sequential([
Dense(80, activation='relu', kernel_initializer=HeNormal(),kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4), input_shape=(X_train.shape[1],)),
BatchNormalization(),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(160, activation='relu', kernel_initializer=GlorotUniform(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)),
BatchNormalization(),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(1, activation='sigmoid', kernel_initializer=RandomNormal(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)) # For binary classification
])
model12.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 80) 3280
batch_normalization (Batch (None, 80) 320
Normalization)
dropout (Dropout) (None, 80) 0
dense_1 (Dense) (None, 160) 12960
batch_normalization_1 (Bat (None, 160) 640
chNormalization)
dropout_1 (Dropout) (None, 160) 0
dense_2 (Dense) (None, 1) 161
=================================================================
Total params: 17361 (67.82 KB)
Trainable params: 16881 (65.94 KB)
Non-trainable params: 480 (1.88 KB)
_________________________________________________________________
#Add the necessary configurations and fit the model
adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
# Customize AdaGrad optimizer
adagrad_optimizer = tf.keras.optimizers.Adagrad(learning_rate=0.01)
model12.compile(optimizer=adagrad_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 25
batch_size = 32
start = time.time()
history = model12.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()
Epoch 1/25 312/312 [==============================] - 3s 5ms/step - loss: 0.5063 - precision: 0.1732 - recall: 0.8964 - val_loss: 0.4447 - val_precision: 0.3110 - val_recall: 0.9028 Epoch 2/25 312/312 [==============================] - 1s 4ms/step - loss: 0.4198 - precision: 0.2149 - recall: 0.8727 - val_loss: 0.3382 - val_precision: 0.4240 - val_recall: 0.8958 Epoch 3/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3685 - precision: 0.2667 - recall: 0.8801 - val_loss: 0.2859 - val_precision: 0.5129 - val_recall: 0.8958 Epoch 4/25 312/312 [==============================] - 2s 8ms/step - loss: 0.3500 - precision: 0.3031 - recall: 0.8652 - val_loss: 0.2550 - val_precision: 0.5677 - val_recall: 0.8958 Epoch 5/25 312/312 [==============================] - 2s 5ms/step - loss: 0.3295 - precision: 0.3433 - recall: 0.8801 - val_loss: 0.2303 - val_precision: 0.6273 - val_recall: 0.8941 Epoch 6/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3162 - precision: 0.3628 - recall: 0.8764 - val_loss: 0.2117 - val_precision: 0.7004 - val_recall: 0.8889 Epoch 7/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3128 - precision: 0.3882 - recall: 0.8783 - val_loss: 0.2017 - val_precision: 0.7288 - val_recall: 0.8958 Epoch 8/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2917 - precision: 0.4247 - recall: 0.8876 - val_loss: 0.1934 - val_precision: 0.7059 - val_recall: 0.8958 Epoch 9/25 312/312 [==============================] - 1s 4ms/step - loss: 0.3079 - precision: 0.4173 - recall: 0.8783 - val_loss: 0.1826 - val_precision: 0.7773 - val_recall: 0.8906 Epoch 10/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2879 - precision: 0.4454 - recall: 0.8783 - val_loss: 0.1761 - val_precision: 0.7743 - val_recall: 0.8993 Epoch 11/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2986 - precision: 0.4215 - recall: 0.8801 - val_loss: 0.1747 - val_precision: 0.7806 - val_recall: 0.8958 Epoch 12/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2733 - precision: 0.4535 - recall: 0.8858 - val_loss: 0.1697 - val_precision: 0.8339 - val_recall: 0.8889 Epoch 13/25 312/312 [==============================] - 2s 6ms/step - loss: 0.2826 - precision: 0.4820 - recall: 0.8764 - val_loss: 0.1686 - val_precision: 0.8100 - val_recall: 0.8958 Epoch 14/25 312/312 [==============================] - 2s 8ms/step - loss: 0.2794 - precision: 0.4668 - recall: 0.8820 - val_loss: 0.1661 - val_precision: 0.7997 - val_recall: 0.9010 Epoch 15/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2689 - precision: 0.4645 - recall: 0.8933 - val_loss: 0.1645 - val_precision: 0.7997 - val_recall: 0.9010 Epoch 16/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2731 - precision: 0.4765 - recall: 0.8933 - val_loss: 0.1613 - val_precision: 0.8122 - val_recall: 0.9010 Epoch 17/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2683 - precision: 0.4917 - recall: 0.8876 - val_loss: 0.1601 - val_precision: 0.8173 - val_recall: 0.9010 Epoch 18/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2737 - precision: 0.4916 - recall: 0.8820 - val_loss: 0.1626 - val_precision: 0.7930 - val_recall: 0.9045 Epoch 19/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2730 - precision: 0.4840 - recall: 0.8764 - val_loss: 0.1585 - val_precision: 0.8201 - val_recall: 0.8941 Epoch 20/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2645 - precision: 0.4958 - recall: 0.8914 - val_loss: 0.1592 - val_precision: 0.7982 - val_recall: 0.9062 Epoch 21/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2552 - precision: 0.4958 - recall: 0.8914 - val_loss: 0.1544 - val_precision: 0.8264 - val_recall: 0.9010 Epoch 22/25 312/312 [==============================] - 1s 5ms/step - loss: 0.2543 - precision: 0.5081 - recall: 0.8858 - val_loss: 0.1593 - val_precision: 0.8278 - val_recall: 0.9010 Epoch 23/25 312/312 [==============================] - 2s 5ms/step - loss: 0.2605 - precision: 0.5080 - recall: 0.8895 - val_loss: 0.1548 - val_precision: 0.8296 - val_recall: 0.8958 Epoch 24/25 312/312 [==============================] - 2s 5ms/step - loss: 0.2514 - precision: 0.4990 - recall: 0.8951 - val_loss: 0.1514 - val_precision: 0.8464 - val_recall: 0.8993 Epoch 25/25 312/312 [==============================] - 1s 4ms/step - loss: 0.2562 - precision: 0.5283 - recall: 0.8914 - val_loss: 0.1508 - val_precision: 0.8296 - val_recall: 0.9045
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model12, X_test, y_test)
print(output)
output_1 = model_performance(model12, X_train, y_train)
print(output_1)
156/156 [==============================] - 0s 1ms/step Accuracy Recall Precision F1 Score 0 0.980758 0.980758 0.981178 0.980946 312/312 [==============================] - 0s 1ms/step Accuracy Recall Precision F1 Score 0 0.98317 0.98317 0.984289 0.983591
results.loc[12] = ['2','80,160','relu,relu',epochs,batch_size,'yes','he/glorot/randnorm','0.3',"Adagrad_LR(0.01)/L1,L2",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
| 4 | 1 | 80 | relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.567311 | 0.193805 | 0.144769 | 0.608861 | 0.707650 | 0.900749 | 0.899306 | 0.973761 | 0.970134 |
| 5 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.894305 | 0.139635 | 0.115674 | 0.714076 | 0.767407 | 0.911985 | 0.899306 | 0.978480 | 0.977551 |
| 6 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(0.01) | 42.051322 | 0.145135 | 0.133539 | 0.785032 | 0.840266 | 0.923221 | 0.876736 | 0.982125 | 0.981760 |
| 7 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(le-5) | 31.487571 | 0.322984 | 0.313230 | 0.305324 | 0.337232 | 0.891386 | 0.901042 | 0.953981 | 0.898577 |
| 8 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | 0.3 | Adam_LR(le-5) | 42.194058 | 0.425188 | 0.440071 | 0.187967 | 0.218922 | 0.848315 | 0.895833 | 0.946604 | 0.818000 |
| 9 | 2 | 80,160 | relu,relu | 25 | 32 | yes | - | 0.3 | Adam_LR(le-5) | 42.708833 | 0.450809 | 0.401692 | 0.145643 | 0.270876 | 0.867041 | 0.923611 | 0.951479 | 0.854279 |
| 10 | 2 | 80,160 | relu,relu | 25 | 32 | yes | he/glorot/randnorm | 0.3 | Adam_LR(le-5) | 37.227174 | 0.436856 | 0.494443 | 0.144936 | 0.217604 | 0.895131 | 0.927083 | 0.948019 | 0.801764 |
| 11 | 2 | 80,160 | relu,relu | 25 | 32 | yes | he/glorot/randnorm | 0.3 | Adam_LR(le-5)/L1,L2 | 42.635221 | 0.467466 | 0.559652 | 0.150646 | 0.213456 | 0.895131 | 0.925347 | 0.946944 | 0.797354 |
| 12 | 2 | 80,160 | relu,relu | 25 | 32 | yes | he/glorot/randnorm | 0.3 | Adagrad_LR(0.01)/L1,L2 | 37.517705 | 0.256188 | 0.150753 | 0.528302 | 0.829618 | 0.891386 | 0.904514 | 0.981178 | 0.980758 |
Model 13 (Model 11 with epoch change)¶
- Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes
- Adam Optimizer - learning_rate = 1e-5
- Loss function - Binary Cross entropy
- Epochs - 50
- Batch size - 32
- Class weights taken into account
- Output layer - Sigmoid Activaion Function
- Drop out - 0.3
- Batch Normalization added
- Weight Initialization done - He (HL1), Glorot (HL2), RandomNormal (Output)
- Regularization - L1 and L2
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
#Initializing the neural network
model13 = Sequential([
Dense(80, activation='relu', kernel_initializer=HeNormal(),kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4), input_shape=(X_train.shape[1],)),
BatchNormalization(),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(160, activation='relu', kernel_initializer=GlorotUniform(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)),
BatchNormalization(),
Dropout(0.3), # Dropout layer with 30% dropout rate
Dense(1, activation='sigmoid', kernel_initializer=RandomNormal(), kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)) # For binary classification
])
model13.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 80) 3280
batch_normalization (Batch (None, 80) 320
Normalization)
dropout (Dropout) (None, 80) 0
dense_1 (Dense) (None, 160) 12960
batch_normalization_1 (Bat (None, 160) 640
chNormalization)
dropout_1 (Dropout) (None, 160) 0
dense_2 (Dense) (None, 1) 161
=================================================================
Total params: 17361 (67.82 KB)
Trainable params: 16881 (65.94 KB)
Non-trainable params: 480 (1.88 KB)
_________________________________________________________________
#Add the necessary configurations and fit the model
adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
model13.compile(optimizer=adam_optimizer,loss='binary_crossentropy',metrics=metrics)
epochs = 50
batch_size = 32
start = time.time()
history = model13.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,validation_data=(X_val,y_val),class_weight=class_weights)
end = time.time()
Epoch 1/50 312/312 [==============================] - 3s 5ms/step - loss: 0.7501 - precision: 0.1531 - recall: 0.7432 - val_loss: 0.7247 - val_precision: 0.1000 - val_recall: 0.7743 Epoch 2/50 312/312 [==============================] - 1s 5ms/step - loss: 0.6894 - precision: 0.0792 - recall: 0.7022 - val_loss: 0.7073 - val_precision: 0.1127 - val_recall: 0.8438 Epoch 3/50 312/312 [==============================] - 1s 5ms/step - loss: 0.6539 - precision: 0.0890 - recall: 0.7715 - val_loss: 0.6970 - val_precision: 0.1222 - val_recall: 0.8837 Epoch 4/50 312/312 [==============================] - 2s 6ms/step - loss: 0.6177 - precision: 0.0930 - recall: 0.8015 - val_loss: 0.6867 - val_precision: 0.1295 - val_recall: 0.8976 Epoch 5/50 312/312 [==============================] - 2s 6ms/step - loss: 0.6159 - precision: 0.0959 - recall: 0.8109 - val_loss: 0.6764 - val_precision: 0.1375 - val_recall: 0.9149 Epoch 6/50 312/312 [==============================] - 2s 5ms/step - loss: 0.5910 - precision: 0.1048 - recall: 0.8446 - val_loss: 0.6666 - val_precision: 0.1446 - val_recall: 0.9236 Epoch 7/50 312/312 [==============================] - 1s 4ms/step - loss: 0.5733 - precision: 0.1065 - recall: 0.8558 - val_loss: 0.6611 - val_precision: 0.1494 - val_recall: 0.9236 Epoch 8/50 312/312 [==============================] - 1s 5ms/step - loss: 0.5597 - precision: 0.1106 - recall: 0.8614 - val_loss: 0.6573 - val_precision: 0.1531 - val_recall: 0.9306 Epoch 9/50 312/312 [==============================] - 1s 5ms/step - loss: 0.5498 - precision: 0.1142 - recall: 0.8820 - val_loss: 0.6501 - val_precision: 0.1581 - val_recall: 0.9323 Epoch 10/50 312/312 [==============================] - 1s 5ms/step - loss: 0.5454 - precision: 0.1151 - recall: 0.8820 - val_loss: 0.6346 - val_precision: 0.1660 - val_recall: 0.9288 Epoch 11/50 312/312 [==============================] - 1s 5ms/step - loss: 0.5373 - precision: 0.1188 - recall: 0.8801 - val_loss: 0.6308 - val_precision: 0.1705 - val_recall: 0.9323 Epoch 12/50 312/312 [==============================] - 1s 4ms/step - loss: 0.5262 - precision: 0.1237 - recall: 0.8914 - val_loss: 0.6257 - val_precision: 0.1741 - val_recall: 0.9340 Epoch 13/50 312/312 [==============================] - 2s 7ms/step - loss: 0.5145 - precision: 0.1281 - recall: 0.9101 - val_loss: 0.6171 - val_precision: 0.1781 - val_recall: 0.9358 Epoch 14/50 312/312 [==============================] - 2s 6ms/step - loss: 0.5129 - precision: 0.1288 - recall: 0.8876 - val_loss: 0.6050 - val_precision: 0.1848 - val_recall: 0.9358 Epoch 15/50 312/312 [==============================] - 1s 5ms/step - loss: 0.5025 - precision: 0.1315 - recall: 0.9064 - val_loss: 0.6070 - val_precision: 0.1840 - val_recall: 0.9340 Epoch 16/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4974 - precision: 0.1374 - recall: 0.9157 - val_loss: 0.5913 - val_precision: 0.1912 - val_recall: 0.9340 Epoch 17/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4973 - precision: 0.1359 - recall: 0.8914 - val_loss: 0.5870 - val_precision: 0.1948 - val_recall: 0.9340 Epoch 18/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4922 - precision: 0.1385 - recall: 0.9007 - val_loss: 0.5784 - val_precision: 0.1999 - val_recall: 0.9340 Epoch 19/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4822 - precision: 0.1414 - recall: 0.9007 - val_loss: 0.5686 - val_precision: 0.2068 - val_recall: 0.9340 Epoch 20/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4749 - precision: 0.1476 - recall: 0.9007 - val_loss: 0.5594 - val_precision: 0.2131 - val_recall: 0.9340 Epoch 21/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4785 - precision: 0.1461 - recall: 0.8951 - val_loss: 0.5551 - val_precision: 0.2165 - val_recall: 0.9323 Epoch 22/50 312/312 [==============================] - 2s 5ms/step - loss: 0.4838 - precision: 0.1461 - recall: 0.8876 - val_loss: 0.5556 - val_precision: 0.2166 - val_recall: 0.9323 Epoch 23/50 312/312 [==============================] - 2s 6ms/step - loss: 0.4701 - precision: 0.1520 - recall: 0.9007 - val_loss: 0.5415 - val_precision: 0.2240 - val_recall: 0.9323 Epoch 24/50 312/312 [==============================] - 2s 5ms/step - loss: 0.4674 - precision: 0.1561 - recall: 0.8989 - val_loss: 0.5277 - val_precision: 0.2348 - val_recall: 0.9323 Epoch 25/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4670 - precision: 0.1575 - recall: 0.8970 - val_loss: 0.5252 - val_precision: 0.2372 - val_recall: 0.9323 Epoch 26/50 312/312 [==============================] - 3s 9ms/step - loss: 0.4608 - precision: 0.1598 - recall: 0.8933 - val_loss: 0.5168 - val_precision: 0.2439 - val_recall: 0.9306 Epoch 27/50 312/312 [==============================] - 6s 18ms/step - loss: 0.4552 - precision: 0.1629 - recall: 0.9007 - val_loss: 0.5190 - val_precision: 0.2399 - val_recall: 0.9306 Epoch 28/50 312/312 [==============================] - 3s 10ms/step - loss: 0.4572 - precision: 0.1678 - recall: 0.9157 - val_loss: 0.5115 - val_precision: 0.2467 - val_recall: 0.9306 Epoch 29/50 312/312 [==============================] - 2s 6ms/step - loss: 0.4579 - precision: 0.1622 - recall: 0.8839 - val_loss: 0.5045 - val_precision: 0.2530 - val_recall: 0.9288 Epoch 30/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4434 - precision: 0.1727 - recall: 0.9139 - val_loss: 0.4904 - val_precision: 0.2659 - val_recall: 0.9271 Epoch 31/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4440 - precision: 0.1758 - recall: 0.9045 - val_loss: 0.4861 - val_precision: 0.2696 - val_recall: 0.9271 Epoch 32/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4365 - precision: 0.1779 - recall: 0.9026 - val_loss: 0.4818 - val_precision: 0.2718 - val_recall: 0.9271 Epoch 33/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4383 - precision: 0.1790 - recall: 0.8989 - val_loss: 0.4802 - val_precision: 0.2736 - val_recall: 0.9271 Epoch 34/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4358 - precision: 0.1876 - recall: 0.8933 - val_loss: 0.4631 - val_precision: 0.2884 - val_recall: 0.9219 Epoch 35/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4268 - precision: 0.1864 - recall: 0.9064 - val_loss: 0.4619 - val_precision: 0.2880 - val_recall: 0.9219 Epoch 36/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4236 - precision: 0.1912 - recall: 0.9026 - val_loss: 0.4649 - val_precision: 0.2841 - val_recall: 0.9219 Epoch 37/50 312/312 [==============================] - 2s 6ms/step - loss: 0.4280 - precision: 0.1911 - recall: 0.8970 - val_loss: 0.4483 - val_precision: 0.3006 - val_recall: 0.9201 Epoch 38/50 312/312 [==============================] - 2s 6ms/step - loss: 0.4193 - precision: 0.2024 - recall: 0.8933 - val_loss: 0.4467 - val_precision: 0.3011 - val_recall: 0.9236 Epoch 39/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4225 - precision: 0.1972 - recall: 0.8933 - val_loss: 0.4433 - val_precision: 0.3042 - val_recall: 0.9236 Epoch 40/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4202 - precision: 0.2069 - recall: 0.9101 - val_loss: 0.4358 - val_precision: 0.3094 - val_recall: 0.9201 Epoch 41/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4156 - precision: 0.2061 - recall: 0.8951 - val_loss: 0.4302 - val_precision: 0.3166 - val_recall: 0.9201 Epoch 42/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4147 - precision: 0.2037 - recall: 0.8914 - val_loss: 0.4212 - val_precision: 0.3245 - val_recall: 0.9184 Epoch 43/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4159 - precision: 0.2069 - recall: 0.8820 - val_loss: 0.4170 - val_precision: 0.3319 - val_recall: 0.9184 Epoch 44/50 312/312 [==============================] - 1s 4ms/step - loss: 0.4129 - precision: 0.2095 - recall: 0.8858 - val_loss: 0.4236 - val_precision: 0.3185 - val_recall: 0.9201 Epoch 45/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4041 - precision: 0.2154 - recall: 0.8989 - val_loss: 0.4161 - val_precision: 0.3284 - val_recall: 0.9184 Epoch 46/50 312/312 [==============================] - 2s 7ms/step - loss: 0.4117 - precision: 0.2160 - recall: 0.8820 - val_loss: 0.4059 - val_precision: 0.3420 - val_recall: 0.9149 Epoch 47/50 312/312 [==============================] - 2s 6ms/step - loss: 0.4050 - precision: 0.2164 - recall: 0.8895 - val_loss: 0.4024 - val_precision: 0.3506 - val_recall: 0.9149 Epoch 48/50 312/312 [==============================] - 1s 4ms/step - loss: 0.3994 - precision: 0.2247 - recall: 0.8820 - val_loss: 0.3970 - val_precision: 0.3539 - val_recall: 0.9149 Epoch 49/50 312/312 [==============================] - 1s 5ms/step - loss: 0.3990 - precision: 0.2276 - recall: 0.8876 - val_loss: 0.3950 - val_precision: 0.3566 - val_recall: 0.9149 Epoch 50/50 312/312 [==============================] - 1s 5ms/step - loss: 0.4033 - precision: 0.2261 - recall: 0.8933 - val_loss: 0.3848 - val_precision: 0.3697 - val_recall: 0.9167
plot_training_history(history,'loss')
plot_training_history(history,'precision')
plot_training_history(history,'recall')
output = model_performance(model13, X_test, y_test)
print(output)
output_1 = model_performance(model13, X_train, y_train)
print(output_1)
156/156 [==============================] - 0s 1ms/step Accuracy Recall Precision F1 Score 0 0.906394 0.906394 0.956138 0.923622 312/312 [==============================] - 0s 1ms/step Accuracy Recall Precision F1 Score 0 0.906231 0.906231 0.959188 0.924512
results.loc[13] = ['2','80,160','relu,relu',epochs,batch_size,'yes','he/glorot/randnorm','0.3',"Adam_LR(le-5)/L1,L2",end-start,history.history['loss'][-1],history.history['val_loss'][-1],
history.history['precision'][-1],history.history['val_precision'][-1],history.history['recall'][-1],history.history['val_recall'][-1]
,output['Precision'][0],output['Recall'][0]]
results
| hidden_layers | neurons_HL | activation_ftn_HL | epochs | batch_size | Batch_Norm | Weight_Ini | dropout | optimizer | time(secs) | Train_loss | Validation_loss | Train_Precision | Validation_Precision | Train_Recall | Validation_Recall | Test_Precision | Test_Recall | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | - | - | - | 10 | 9982 | - | - | - | SGD | 1.234703 | 2.031547 | 1.695963 | 0.054343 | 0.056860 | 0.219101 | 0.189236 | 0.890782 | 0.764682 |
| 1 | - | - | - | 25 | 9982 | - | - | - | SGD_LR(0.01) | 3.018212 | 0.425082 | 0.430649 | 0.402844 | 0.445483 | 0.318352 | 0.248264 | 0.928051 | 0.940469 |
| 2 | - | - | - | 50 | 9982 | - | - | - | SGD-LR(0.01) | 2.985321 | 0.434529 | 0.493189 | 0.566038 | 0.598456 | 0.280899 | 0.269097 | 0.942613 | 0.951694 |
| 3 | - | - | - | 25 | 32 | - | - | - | SGD-LR(0.01) | 27.868973 | 2.379025 | 2.360701 | 0.022727 | 0.023452 | 0.046816 | 0.043403 | 0.888372 | 0.844257 |
| 4 | 1 | 80 | relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.567311 | 0.193805 | 0.144769 | 0.608861 | 0.707650 | 0.900749 | 0.899306 | 0.973761 | 0.970134 |
| 5 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | SGD_LR(0.01) | 41.894305 | 0.139635 | 0.115674 | 0.714076 | 0.767407 | 0.911985 | 0.899306 | 0.978480 | 0.977551 |
| 6 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(0.01) | 42.051322 | 0.145135 | 0.133539 | 0.785032 | 0.840266 | 0.923221 | 0.876736 | 0.982125 | 0.981760 |
| 7 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | - | Adam_LR(le-5) | 31.487571 | 0.322984 | 0.313230 | 0.305324 | 0.337232 | 0.891386 | 0.901042 | 0.953981 | 0.898577 |
| 8 | 2 | 80,160 | relu,relu | 25 | 32 | - | - | 0.3 | Adam_LR(le-5) | 42.194058 | 0.425188 | 0.440071 | 0.187967 | 0.218922 | 0.848315 | 0.895833 | 0.946604 | 0.818000 |
| 9 | 2 | 80,160 | relu,relu | 25 | 32 | yes | - | 0.3 | Adam_LR(le-5) | 42.708833 | 0.450809 | 0.401692 | 0.145643 | 0.270876 | 0.867041 | 0.923611 | 0.951479 | 0.854279 |
| 10 | 2 | 80,160 | relu,relu | 25 | 32 | yes | he/glorot/randnorm | 0.3 | Adam_LR(le-5) | 37.227174 | 0.436856 | 0.494443 | 0.144936 | 0.217604 | 0.895131 | 0.927083 | 0.948019 | 0.801764 |
| 11 | 2 | 80,160 | relu,relu | 25 | 32 | yes | he/glorot/randnorm | 0.3 | Adam_LR(le-5)/L1,L2 | 42.635221 | 0.467466 | 0.559652 | 0.150646 | 0.213456 | 0.895131 | 0.925347 | 0.946944 | 0.797354 |
| 12 | 2 | 80,160 | relu,relu | 25 | 32 | yes | he/glorot/randnorm | 0.3 | Adagrad_LR(0.01)/L1,L2 | 37.517705 | 0.256188 | 0.150753 | 0.528302 | 0.829618 | 0.891386 | 0.904514 | 0.981178 | 0.980758 |
| 13 | 2 | 80,160 | relu,relu | 50 | 32 | yes | he/glorot/randnorm | 0.3 | Adam_LR(le-5)/L1,L2 | 83.503170 | 0.403283 | 0.384788 | 0.226066 | 0.369748 | 0.893258 | 0.916667 | 0.956138 | 0.906394 |
Actionable Insights & Buisiness Recommendations¶
Actionable Insights:
- The model with best performance was Model13 with Recall metrics of importance as follows,
Testing - Recall - 0.906394
Training - Recall - 0.906231
- Configuration used for Model13 is as follows,
- Two Hidden layer - ReLu activation function - 80 nodes, 160 nodes, Output layer - Sigmoid Activaion Function
- Adam Optimizer - learning_rate = 1e-5
- Loss function - Binary Cross entropy
- Epochs - 50 / Batch size - 32
- Class weights taken into account
- Drop out - 0.3 / Batch Normalization added
- Weight Initialization done - He (HL1), Glorot (HL2), RandomNormal (Output)
- Regularization - L1 and L2
Started working with shallow Networks and then worked my way by increasing the epochs first and then batch size. Increasing the epoch got the smoothness to the metrics curve.
Adam Optimizer proved to be a better choice compared to standard SGD and Adagrad. Adam with a slower learning rate worked better.
Other techniques like Batch normalization, dropout of 30%, various weight initialization and L1,L2 regularization techniques helped a lot in improving and smoothening the gap between the Training and Testing Metrics.
Buisiness Recommendations:
- Features V7,V11,V15,V16,V21,V28,V34 have the most impact on determining whether a windmill will fail or not.¶
- The company should focus on improving the important features so they can reduce the number of failures and the amount of money spent on repairs and replacements.